Fix bugs and Improve the approach of makemovie #900
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
1. fix the bugs originated from the latest commit b544241
The commit b544241 made a modification to the keras.py part on which the makemovie.py heavily depends. This results in errors of the code calling the keras part,
donkeycar/donkeycar/management/makemovie.py
Line 76 in 50f0831
donkeycar/donkeycar/management/makemovie.py
Lines 147 to 148 in 50f0831
donkeycar/donkeycar/management/makemovie.py
Line 207 in 50f0831
2. improve the approach to calculate saliency maps,
2.1 evaluate the effects of all output layers instead of one
2.2 correct the problematic math (blending the gradients of angle and throttle output layers together) from direct sum to square norm
2.3 calculate the saliency depending on the model type "linear" and "categorical"
3. modify the "draw_user_input", "draw_model_prediction", and "draw_steering_distribution" methods
I add an argument taking an image array to each method, such that the labels are drawn on this image array instead of the same image for inference. For instance, if we draw the user input green line on the input image, and then pass the same image for inference, that green line could affect the model prediction and give an incorrect prediction blue line.

These labels (yellow-purple cloud, blue line, green line, red line and grey lines) could affect the model prediction. It is better to seperate the image array for inference and the image array on which labels are drawn.