Add ability to set custom distance function for counterfactuals #2607

tolga-b · 2019-08-29T18:46:05Z

Motivation for features / changes
WIT did not support counterfactual computation for image and text features as it is not straightforward to define the similarity of two images or two sentences. This PR adds the ability to pass a custom python function for similarity computation such that WIT can query that function to compute counterfactuals. This allows the flexibility for users to define and experiment with their custom similary metrics and opens the option to load even a separate deep learning model for similarity purposes.
In addition, smile detection demo is updated to use tfjs MobileNet model embeddings for example distances, allowing counterfactuals using image features (previously this demo only used tabular data for counterfactuals while smile detection model only used the image.)
Technical description of changes
WitConfig now has set_custom_distance_fn to pass a python function.
If a custom distance function is not set, WIT behaves the same as before (L1/L2 distance).
Boilerplate code to request distances from js side and call a js callback function from the python side when the distances are ready.
Screenshots of UI changes
If custom distance function is set, L1 L2 distance selection is greyed out and a third option "User-specified" appears. The same is true for show similarity to selected datapoint option.
Counterfactuals:

Show similarity to selected datapoint pane:

Facets scatterplot w.r.t MobileNet embedding distance:
Detailed steps to verify changes work correctly (as executed by you)
Verified jupyter notebook and colab notebook usage by creating a Smile detection demo with custom distance method that uses the difference in average image color for distance between two images.
Ran all browser demos, including the new smile detection demo that uses custom distance function.
Alternate designs / implementations considered

tolga-b · 2019-08-29T19:28:56Z

@jameswex I will do one more pass on this to check if everything works correctly (demos etc.). addDistance with custom distance function will have it's own PR. You can start checking it now or after that pass, thank you!

mpushkarna · 2019-08-29T20:33:59Z

@tolga-b cc:@jameswex
For a complete UI implementation, when users set a custom distance function:
(a) If users can only use the custom distance metric, radio buttons for L1 or L2 distances should be disabled or
(b) Users should be able to pick from L1, L2 and custom distance in a dropdown.

jameswex

how much work would it be to add this for adding distance feature as well? want in separate PR?

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

tensorboard/plugins/interactive_inference/witwidget/notebook/base.py

tensorboard/plugins/interactive_inference/witwidget/notebook/colab/wit.py

tensorboard/plugins/interactive_inference/witwidget/notebook/jupyter/wit.py

tensorboard/plugins/interactive_inference/witwidget/notebook/visualization.py

jameswex · 2019-08-29T21:34:00Z

@tolga-b cc:@jameswex
For a complete UI implementation, when users set a custom distance function:
(a) If users can only use the custom distance metric, radio buttons for L1 or L2 distances should be disabled or
(b) Users should be able to pick from L1, L2 and custom distance in a dropdown.

I think if users set their own custom distance measure that we can always use that and remove the controls for l1 and l2, as in these cases the auto-computed measure doesn't make sense (like we currently see in our text and image examples).

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

jameswex · 2019-09-04T00:06:14Z

what is left to get this fully working with appropriate UI? if nothing, then can you also update the smiling demo and a notebook demo with custom distance so it can be tested in this PR? Thanks.

…o distance

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

…to distance

…o distance

tensorboard/plugins/interactive_inference/README.md

...e_inference/tf_interactive_inference_dashboard/demo/tf-interactive-inference-image-demo.html

wchargin · 2019-09-12T18:25:31Z

(I cancelled this build because the first two jobs have lint errors, so
the rest would have failed in the same way, and I’d like to unblock CI
for #2651, which blocks a 1.15 release. Hope you don’t mind.)

…to distance

jameswex · 2019-09-13T13:21:43Z

@stephanwlee I have reviewed this code. would you or another TB team member also like to before it is merged?

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

tensorboard/plugins/interactive_inference/witwidget/notebook/base.py

third_party/js.bzl

stephanwlee · 2019-09-17T15:33:53Z

tensorboard/plugins/interactive_inference/witwidget/notebook/jupyter/js/lib/wit.js

    this.inferMutantsCounter = 0;
    this.view_.addEventListener('infer-mutants', (e) => {
      e.detail['infer_mutants_counter'] = this.inferMutantsCounter++;
      this.model.set('infer_mutants', e.detail);
      this.mutantFeature = e.detail.feature_name;
      this.touch();
    });
+    this.computeDistanceCounter = 0;
+    this.view_.addEventListener('compute-custom-distance', (e) => {
+      e.detail['compute_distance_counter'] = this.computeDistanceCounter++;


this feels quite odd and not clean. Would it be possible to use data binding instead?

The counter is there as it is in the case of infer-mutants above. We use data binding for what is passed ('compute_custom_distance' is set to e.detail). However, we only pass the example id back to the python side instead of the full example since examples are already in sync between the backend and frontend. In this case, if someone requests the same example again after updating some features, the change on bound dict may not trigger since it is the same dict. We force trigger it by incrementing the counter.

Sorry but I must confess I am very lost. Even if this logic is correct, I do think this has readability problem.

AFAICT, the event is fired from JavaScript side. When firing the event, the object does not contain the property compute_distance_counter. 1. how is it being used by owner of the event object? 2. where does the Python come into play?

The dashboard launches this event from js side by passing some parameters (example index, callback ind etc.). "compute_custom_distance" is sync variable between js and python sides. This synchronization is managed by jupyter widget framework. When we call this.model.set('compute_custom_distance', e.detail), ideally jupyter should recognize the change in "compute_custom_distance" and trigger distance computation. James and I found out previously (for infer-mutants) that if someone edits an example then fires this event again from the dashboard, since e.detail does not contain the contents of the example but only it's index, jupyter thinks nothing has changed and model.set does not trigger distance computation. James solved it by adding a counter so that we make sure the serialized dictionary always has one field that changes between calls and jupyter triggers the sync.
Since this is a jupyter specific issue, we did not include this in the event but append it in wit.js related to jupyter.

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

third_party/js.bzl

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

stephanwlee · 2019-09-17T21:00:32Z

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

+            callback: callbackFun,
+            params: params,
+          };
+          this.fire('compute-custom-distance', urlParams);


I find it weird to see notebook specific code in the main dashboard code. What is the expected behavior outside of the notebook?

requestDistanceWithCallback would not be invoked in non-local instances (local=demos and notebook). In case we are in non-local mode, WIT defaults to it's previous behavior of computing counterfactuals completely on the js side with L1 and L2 distance between examples. This is slightly similar in terms of behavior to custom_predict_fn where it is only supported in notebook mode.

Added TODO note here so that if there is a support in TensorBoard mode to provide custom distance functions for counterfactuals then we should update this function to reflect that.

tolga-b · 2019-09-18T19:55:37Z

Thank you, @stephanwlee I addressed the review comments. Could you please check it out?

third_party/js.bzl

stephanwlee · 2019-09-18T20:41:34Z

tensorboard/plugins/interactive_inference/witwidget/notebook/jupyter/js/lib/wit.js

    this.inferMutantsCounter = 0;
    this.view_.addEventListener('infer-mutants', (e) => {
      e.detail['infer_mutants_counter'] = this.inferMutantsCounter++;
      this.model.set('infer_mutants', e.detail);
      this.mutantFeature = e.detail.feature_name;
      this.touch();
    });
+    this.computeDistanceCounter = 0;
+    this.view_.addEventListener('compute-custom-distance', (e) => {
+      e.detail['compute_distance_counter'] = this.computeDistanceCounter++;


Sorry but I must confess I am very lost. Even if this logic is correct, I do think this has readability problem.

AFAICT, the event is fired from JavaScript side. When firing the event, the object does not contain the property compute_distance_counter. 1. how is it being used by owner of the event object? 2. where does the Python come into play?

tolga-b · 2019-09-18T22:29:47Z

Thank you Stephan, I removed the side effect in wit.js event object per your suggestion!

tensorboard/plugins/interactive_inference/witwidget/notebook/jupyter/js/lib/wit.js

tolga-b added 6 commits August 22, 2019 22:17

midway 1

ae2daff

midway 2

d6a6920

midway 3

0359655

image colab works

fc6709b

colab distance 1

31e05ae

jupyter works

79dd89b

googlebot added the cla: yes label Aug 29, 2019

ran prettier

00cade8

jameswex reviewed Aug 29, 2019

View reviewed changes

tolga-b added 3 commits September 3, 2019 18:25

review comments

ed3a4c6

minor fix

299dd47

fix lint

76ddff8

jameswex reviewed Sep 4, 2019

View reviewed changes

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html Outdated Show resolved Hide resolved

tolga-b added 2 commits September 3, 2019 17:41

fix comment

0ee315a

Merge branch 'distance' of https://github.com/tolga-b/tensorboard int…

a5126e8

…o distance

jameswex reviewed Sep 4, 2019

View reviewed changes

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html Outdated Show resolved Hide resolved

tolga-b added 6 commits September 3, 2019 17:59

fix dom if

9749fd6

image demo 1

1343f30

image demo 2

cc2ab68

image demo 3

3fb28c1

image demo 3

47b5960

undo image demo

243e78e

tolga-b mentioned this pull request Sep 5, 2019

Add custom distance for smile detector demo PAIR-code/what-if-tool#17

Merged

tolga-b added 3 commits September 5, 2019 19:56

Merge branch 'master' of https://github.com/tensorflow/tensorboard in…

4eb6e09

…to distance

Merge branch 'distance' of https://github.com/tolga-b/tensorboard int…

9e98468

…o distance

add distance

674afd4

jameswex reviewed Sep 12, 2019

View reviewed changes

tensorboard/plugins/interactive_inference/README.md Outdated Show resolved Hide resolved

jameswex reviewed Sep 12, 2019

View reviewed changes

...e_inference/tf_interactive_inference_dashboard/demo/tf-interactive-inference-image-demo.html Outdated Show resolved Hide resolved

tolga-b added 2 commits September 12, 2019 12:56

update demos

d2414d9

review comments fix image demo slice

0c4a9c9

tolga-b added 4 commits September 12, 2019 14:26

memory leak fix

2469392

memory leak fix 2

859cfa6

ran prettier

87eb749

Merge branch 'master' of https://github.com/tensorflow/tensorboard in…

d38f7b5

…to distance

jameswex approved these changes Sep 12, 2019

View reviewed changes

smile demo upload image fix

50839a3

jameswex requested a review from stephanwlee September 16, 2019 17:59

stephanwlee reviewed Sep 17, 2019

View reviewed changes

tolga-b added 2 commits September 17, 2019 14:28

review comments

7c82372

merge master

9aa1a72

stephanwlee reviewed Sep 17, 2019

View reviewed changes

tolga-b added 6 commits September 18, 2019 15:26

review comments

8b02c04

prettier

de0ca1b

merge master

69f5d6d

fix bzl

b87964c

fix bzl 2

5becf6a

fix bzl 3

a80af0f

stephanwlee reviewed Sep 18, 2019

View reviewed changes

review comments

13a8444

jameswex mentioned this pull request Sep 19, 2019

Fix counterfactual model dropdown #2680

Closed

stephanwlee approved these changes Sep 19, 2019

View reviewed changes

tensorboard/plugins/interactive_inference/witwidget/notebook/jupyter/js/lib/wit.js Show resolved Hide resolved

jameswex merged commit 15456be into tensorflow:master Sep 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to set custom distance function for counterfactuals #2607

Add ability to set custom distance function for counterfactuals #2607

tolga-b commented Aug 29, 2019 •

edited

Loading

tolga-b commented Aug 29, 2019

mpushkarna commented Aug 29, 2019

jameswex left a comment

jameswex commented Aug 29, 2019

jameswex commented Sep 4, 2019

wchargin commented Sep 12, 2019

jameswex commented Sep 13, 2019

stephanwlee Sep 17, 2019

tolga-b Sep 17, 2019

stephanwlee Sep 18, 2019

tolga-b Sep 18, 2019

stephanwlee Sep 17, 2019

tolga-b Sep 17, 2019

tolga-b Sep 18, 2019

tolga-b commented Sep 18, 2019

stephanwlee Sep 18, 2019

tolga-b commented Sep 18, 2019

Add ability to set custom distance function for counterfactuals #2607

Add ability to set custom distance function for counterfactuals #2607

Conversation

tolga-b commented Aug 29, 2019 • edited Loading

tolga-b commented Aug 29, 2019

mpushkarna commented Aug 29, 2019

jameswex left a comment

Choose a reason for hiding this comment

jameswex commented Aug 29, 2019

jameswex commented Sep 4, 2019

wchargin commented Sep 12, 2019

jameswex commented Sep 13, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tolga-b commented Sep 18, 2019

Choose a reason for hiding this comment

tolga-b commented Sep 18, 2019

tolga-b commented Aug 29, 2019 •

edited

Loading