fix(ntp_monitor): move chronyc command execution to a timer

[ito-san]

Description

The diagnostic topic of ntp_monitor sometimes may not be sent periodically, every 1 second.
The root cause is that the execution of the chronyc tracking command gets stuck when the network connection is poor.

So the execution of the chronyc will be separated from the main thread using a separate thread along with a timer.
In addition, a feature will be added to monitor timeouts in chronyc and to send a WARNING notification when a timeout occurs.
The timeout duration is set to 5 seconds by default.

Related links

TIER IV INTERNAL LINK TO JIRA

Tests performed

1. Run Autoware.

2. Run rqt_runtime_monitor.

   ros2 run rqt_runtime_monitor rqt_runtime_monitor

3. Verify OK is reported during normal operation.
   

4. Verify Warning is reported when chronyc tracking gets stuck.
   

Notes for reviewers

None

Interface changes

None

Effects on system behavior

None

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

• I've confirmed the contribution guidelines.
• The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

• The PR follows the pull request guidelines.
• The PR has been properly tested.
• The PR has been reviewed by the code owners.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

• There are no open discussions or they are tracked via tickets.
• The PR is ready for merge.

After all checkboxes are checked, anyone who has write access can merge the PR.

[codecov]

Codecov Report

Patch coverage has no change and project coverage change: -0.25% ⚠️

Comparison is base (bf03fe3) 15.02% compared to head (93ca423) 14.77%.
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4634      +/-   ##
==========================================
- Coverage   15.02%   14.77%   -0.25%     
==========================================
  Files        1609     1546      -63     
  Lines      111684   106282    -5402     
  Branches    34605    32472    -2133     
==========================================
- Hits        16784    15707    -1077     
+ Misses      76201    73338    -2863     
+ Partials    18699    17237    -1462     

Flag	Coverage Δ		*Carryforward flag
differential	0.00% <0.00%> (?)
total	14.78% <ø> (-0.25%)	⬇️	Carriedforward from 4c33a4a

*This pull request uses carry forward flags. Click here to find out more.

Files Changed	Coverage Δ
...ion/ndt_scan_matcher/src/ndt_scan_matcher_core.cpp	0.00% <ø> (ø)
...tem/system_monitor/src/ntp_monitor/ntp_monitor.cpp	0.00% <0.00%> (ø)

... and 419 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[h-ohta]

Could you add parameter to param.yaml with detail comments?
https://github.com/autowarefoundation/autoware.universe/blob/4c33a4a3011a2ac840922ead44d5dce5f14fcf99/system/system_monitor/config/ntp_monitor.param.yaml

[ito-san]

Could you add parameter to param.yaml with detail comments? https://github.com/autowarefoundation/autoware.universe/blob/4c33a4a3011a2ac840922ead44d5dce5f14fcf99/system/system_monitor/config/ntp_monitor.param.yaml

@h-ohta
You mean the newly added parameter timeout?

[ito-san]

@h-ohta
You mean the newly added parameter timeout?

@h-ohta
Would you please answer the question?

[h-ohta]

@ito-san Yes

[ito-san]

@ito-san Yes

@h-ohta Fixed in 94d4e90

[h-ohta]

I checked this PR with following procedures.

1. Run system_monitor with ros2 launch

$ ros2 launch system_monitor system_monitor.launch.xml # for working in multi process

1. Run rqt_runtime_monitor.

$ ros2 run rqt_runtime_monitor rqt_runtime_monitor

1. Confirm during normal operation

1. Confirm during abnormal operation, that is command stucking

I changed the command chronyc tracking to sleep 10.

autoware.universe/system/system_monitor/src/ntp_monitor/ntp_monitor.cpp

Line 188 in 93ca423

bp::child c("chronyc tracking", bp::std_out > is_out);

And I found out that ntp_monitor turned to warning state.

[ito-san]

@asana17 Could you review this PR as the code owner?

[asana17]

LGTM.