Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bugs and Improve the approach of makemovie #900

Merged
merged 1 commit into from
Jul 21, 2021

Conversation

BillyCheung10botics
Copy link
Contributor

1. fix the bugs originated from the latest commit b544241

The commit b544241 made a modification to the keras.py part on which the makemovie.py heavily depends. This results in errors of the code calling the keras part,

self.do_salient = self.init_salient(self.keras_part.model)
pred_img = pred_img.reshape((1,) + pred_img.shape)
angle_binned, _ = self.keras_part.model.interpreter.predict(pred_img)
expected = self.keras_part.model.inputs[0].shape[1:]

2. improve the approach to calculate saliency maps,

2.1 evaluate the effects of all output layers instead of one
2.2 correct the problematic math (blending the gradients of angle and throttle output layers together) from direct sum to square norm
2.3 calculate the saliency depending on the model type "linear" and "categorical"

3. modify the "draw_user_input", "draw_model_prediction", and "draw_steering_distribution" methods

I add an argument taking an image array to each method, such that the labels are drawn on this image array instead of the same image for inference. For instance, if we draw the user input green line on the input image, and then pass the same image for inference, that green line could affect the model prediction and give an incorrect prediction blue line.
Screenshot from 2021-07-13 18-28-53
These labels (yellow-purple cloud, blue line, green line, red line and grey lines) could affect the model prediction. It is better to seperate the image array for inference and the image array on which labels are drawn.

1.  fix the bugs originated from the latest commit b544241 (Add support for all other (except KerasLatent) models: (autorope#884))

2.  improve the approach for calculating saliency maps,
2.1 evaluate the effects of all output layers instead of one
2.2 correct the problematic math (blending the gradients of angle and throttle output layers together) from direct sum to square norm
2.3 calculate the saliency depending on the model type "linear" and "categorical"

3.  modify the "draw_user_input", "draw_model_prediction", and "draw_steering_distribution" methods, seperate the image array for inference and the image array that labels are drawn on
naisy added a commit to naisy/donkeycar that referenced this pull request Jul 14, 2021
@sctse999
Copy link
Contributor

@DocGarbanzo @naisy any further comment? If no I will merge this.

Copy link
Contributor

@DocGarbanzo DocGarbanzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for that contribution - Looks good to me. Just a minor comment.

@@ -204,7 +215,7 @@ def draw_salient(self, img):

alpha = 0.004
beta = 1.0 - alpha
expected = self.keras_part.model.inputs[0].shape[1:]
expected = self.keras_part.interpreter.model.inputs[0].shape[1:]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a member function get_input_shapes() in KerasPilot.

@sctse999 sctse999 merged commit 5e2c560 into autorope:dev Jul 21, 2021
@BillyCheung10botics BillyCheung10botics deleted the fixmakemovie branch July 21, 2021 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants