Add playback event recorder

[iche033]

Depends on gazebosim/gz-sim#419 and gazebosim/gz-common#105

The playback event recorder creates videos for events that are stored in the events.yml log file in the state log directory.

Usage:

bash record_playback_events.bash <path_to_log_dir>

The script launches ign gazebo in playback mode and creates videos using the gui camera video recording feature. For each event, it seeks playback to some time before the event, moves the camera to the desired location (either in follow mode or moves to a static pose), and starts recording until some time after the event. After recording all events, ign -gazebo will exit on its own and you can find the recorded videos in a timestamped directory inside the directory where the bash script is run

You can configure the gui camera behavior. To use static gui camera pose during recording, set PlaybackEventRecorder system's <camera_follow> sdf param to false. Currently it defaults to true, which means it'll follow the robot when playing back the event.

Here are the events being recorded and associated the camera placement:

• robots exiting staging area: static camera above staging area looking down at angle
• rock fall: static camera above dynamic rocks model looking directly downward
• marsupial vehicle detaching child robot: camera follows the child robot
• artifact proximity: camera follows the robot
• region of interest: camera follows the robot

[iche033]

updates:

• now recording rock fall events 9d532d9,
• merged detector events that are less than 60 seconds apart (e.g. if the same robot enters then exits the proximity of an artifact within 60s, it will now be captured in the same video recording) 90f2a7d. no longer needed

[iche033]

more changes in 49cf846. Camera placement is now automatically based on event type. Static cameras for detector and rock fall events, while robot follow mode in marsupial vehicle detach evets.

[osrf-jenkins]

Build finished. No test results found.

[peci1]

Hi Ian, I'm experimenting with this recorder - thanks very much, it looks really useful. However, I'm having problems with the commit that added feedback from the recorder stats. It seems that on my system, nothing publishes them. Does it need some custom build of any other libraries? I'm using a dockerfile modified from cloudsim_sim image, but which checks out this branch. So it should use Ignition Citadel and all the correct versions of everything (if they're correctly declared).

Thanks for help!

[peci1]

And I have a few remarks possibly worth considering:

1. The initial event recording staging area goes pretty bad with our deployment strategy. Our last UAV starts 15 minutes before mission end, so the initial staging area recording would be like 50 minutes long and practically useless. Would there be a way to either expose an option for this, or move this to be the last event processed (so that I could stop the recorder when all other events are done)? Or maybe a duration limit could be added for this event?
2. Please consider applying the following patch to the bash script that launches the recorder. It allows running the recorder even in the case that the source tree of subt_ign is on a readonly filesystem (which is the case when we convert the Docker image to Singularity for use on our cluster). It changes the behavior so that the script can be run from any directory and the temporary files are stored in that directory.

--- a/subt_ign/scripts/playback_event_recorder/record_playback_events.bash	(revision 4cc98ea08cf993f56e11387db21d0eaa885b7852)
+++ b/subt_ign/scripts/playback_event_recorder/record_playback_events.bash	(date 1603567761213)
@@ -24,7 +24,7 @@
 fi
 
 scriptDir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-tmpDir="$scriptDir/tmp_record"
+tmpDir="tmp_record"
 
 echo "Creating tmp dir for recording: $tmpDir"
 
@@ -38,7 +38,7 @@
 
 export IGN_GAZEBO_SYSTEM_PLUGIN_PATH=$LD_LIBRARY_PATH
 sdfName="playback_event_recorder"
-ign gazebo -v 4 $sdfName.sdf
+ign gazebo -v 4 "${scriptDir}/${sdfName}.sdf"
 
 echo "Video recording ended. Shutting down playback"

[peci1]

And here are my additions to path_tracer (also a very useful one, probably even more than the event recorder):

Bug fixes:

1. peci1@79fd034
2. peci1@5e6335a

Features:

1. Breadcrumb rendering peci1@0422235
2. Print colors corresponding to each robot peci1@9cc70df
3. Render current robot pose more visibly peci1@506c13b
4. "Dim" older parts of the trajectories peci1@9846b5b
5. Better marker sizes peci1@ed4e42e

And here's a script I use to launch the path_tracer to automatically create a video: https://gist.github.com/peci1/501ad6decba37e1f72eeb31898fc3aad . It automatically records a video, autoconfigures the top-view camera to view exactly the explored space, zooms out several levels in the end to show the whole world, and repeats the last frames of the video a few times for easier inspection of what the state was in the very end.

Feel free to cherry-pick any of my changes ;)

[osrf-jenkins]

Build finished. No test results found.

[iche033]

However, I'm having problems with the commit that added feedback from the recorder stats. It seems that on my system, nothing publishes them. Does it need some custom build of any other libraries

yeah forgot to add the related branches in this PR description. I just updated to say it depends on gazebosim/gz-sim#419 and gazebosim/gz-common#105. The changes in these PR ensure that we can generate videos of higher quality (by setting bitrate) and also correct timing. Make sure to add the necessary fields in ~/.ignition/gazebo/gui.config

1. The initial event recording staging area goes pretty bad with our deployment strategy.

hmm we are generating videos based on a filtered list of events, i.e. modified events.yml, so one workaround is to modify the events.yml and remove the ones associated with staging_area? If that does not work for you, I think one easy change I can do is to add it to the back of the list like you mentioned.

Please consider applying the following patch to the bash script that launches the recorder.

yep done. df35512

[iche033]

And here are my additions to path_tracer (also a very useful one, probably even more than the event recorder)

thanks! I believe @nkoenig is reworking the path tracer video generation script. I'll let him decide how to best incorporate these suggested changes.

[peci1]

hmm we are generating videos based on a filtered list of events

If it makes sense, this filtering can be added to the launch script.

add it to the back of the list like you mentioned

At least like that... But I don't see the reason why this recording is there. All robots will have a detect event from the tile right next to the staging area, and the recording goes back about a minute, so each robot will have a recording of it leaving the staging area. But it won't have the static camera pointed at all robots together, which I can imagine could be a reason to make the staging area recording...

Also, when the lockstepping is finished, do you consider adding an option to run the event playback super-realtime?

[osrf-jenkins]

Build finished. No test results found.

[iche033]

the staging event is now recorded at the end

As for event filtering, that's done externally and given to us to generate the videos. Similarly, the staging area video recording and camera pose is a specific request that came directly from the organizers :)

[iche033]

Also, when the lockstepping is finished, do you consider adding an option to run the event playback super-realtime?

Once the first video recording starts, there are around 15 real time seconds in between the subsequent recordings (with sim mostly paused) to wait for models and levels to properly load on the gui side so I try not to speed that up in case we're doing video recording on less powerful machines.

[osrf-jenkins]

Build finished. No test results found.

[osrf-jenkins]

Build finished. No test results found.

[nkoenig]

This now contains all the code used to create Cave Circuit videos.

[osrf-jenkins]

Build finished. No test results found.

[osrf-jenkins]

Build finished. No test results found.

[peci1]

I've updated edifice branch to build with the current status of Ignition libraries - targeted sdformat11 and transport10, and the subt workspace built successfully. However, if I run path_tracer from the edifice branch, it gets stuck at the very beginning - pausing the simulation to catch up with the video recorder. The problem is that the video recorder stops publishing any updates on the stats topic as soon as the simulation gets paused. And that's a deadlock because the path tracer waits for the recorder stats to arrive with better information.

I guess this did not happen earlier, so what could be the reason for this behavior?

I've even built ign-common3 from source to get the HW-accelerated video encoding, but it still suffers from this problem. It seems to me that the video recorder publishes the stats wrongly and is faster than it says on the stats topic. The simulation always stops at sim time 69 with the video recorder telling that it has encoded 43 seconds of video. But that's weird because I think the encoder should be faster than the simulation (at least earlier it was for 8 Mbps FullHD).