Image processing component - clicking on arbitrary coordinates of images

[farleylai]

Dash looks like the bridge between GUI and visualization. But how well does Dash (plan) support raw data such as images, audio and even video?

I recently build ML models to train over image dataset. To validate the trained models, I would like to have a GUI app that allows the user to inspect image patches. So the app must capture the mouse click coordinates and feed the corresponding image patch to the model. The output image data produced by the model should be rendered on the fly along with some other computed statistics (plots or numbers). I know the HTML img tag has an attribute ismap to capture this. Is that well supported in Dash or are there some component widgets already available for those use cases such as image filtering, selecting a range of audio wave or video timeline to play and so on?

Afterthought

After going through the guide and the core/html component libraries, high level components for non-text and non-graph data are not out of the box but may be imported from other existing react components (react-cursor-position, react-wavesurfer, ...). It seems like the React component sources must be ready for customization. Is it really necessary? Why not build a platform-agnostic GUI abstraction in Python and generate the React sources only for testing and deployment time? In this sense, a web SPA is simply a target platform which the developer may not even be aware of. In the future, other mobile platforms may be supported, not to mention other potential JS UI component libraries.

[ned2]

As you observed, you would indeed need to achieve this kind of functionality through a custom React component, either by making an existing one compatible with Dash or by creating your own. There is a guide to writing your own components here. This would definitely make a pretty useful component if you or someone else were to port/create one!

At the end of the day, you have to target some kind of GUI framework for supporting the creation of arbitrary user interfaces. Dash happens to have chosen the web platform, and more specifically, React. For many Dash apps, developers already can be ignorant of the React implementation details -- that's one of Dash's main selling points. But if your needs go beyond the current Dash components, then yep, you gotta get your hands dirty with some Javascript.

[farleylai]

I respect your design decision though getting rid of the dependency on a particular JS UI lib may seem more appealing. Nonetheless, I figured out a short term solution using matplotlib on Jupyter by calling fig.canvas.mpl_connect() to connect mouse click events which expose the subplot axis and pixel coordinates on click. I noticed that your APIs show images by setting the layout and the GraphWidget accepts on_click events. Can it be used in a similar way as matplotlib on jupyter and expose equivalent event information?

[ned2]

I'm not the designer; I just really like Dash.

Currently, Dash Components support triggering callbacks with click events and also keep track of an n_clicks property, which counts the number of times the component has been clicked. That, I think, is the extent to which mouse-related interaction is supported on Dash's HTML components.

[farleylai]

Alright, I get it. Unfortunately, n_clicks is insufficient in my use case.

[ned2]

Now that I think about it, for getting image selection data, one possibility could be to target the Graph component's relayoutData property, which is updated with data pertaining to the current selection, as illustrated here, and then overlay an image on top of the graph as described here.

The issues I see with this are 1) that synchronising the dimensions of the chart with the image position data looks like it will be awkward and 2) that this seems to only work when dragmode is set to 'zoom', which means your scale is going to be zoomed after every selection.

I suspect re-purposing a React image selection component might be a more promising approach, but wait for @chriddyp to weigh in on this.

[chriddyp]

Why not build a platform-agnostic GUI abstraction in Python and generate the React sources only for testing and deployment time?
though getting rid of the dependency on a particular JS UI lib may seem more appealing

Dash is modular - the Python dash backend only serializes python components as JSON. You could write a different front-end that communicates with the same HTTP endpoints but renders the JSON representation of the components using different technology. Here is the current implementation of the Dash front-end (in React) if you are curious: https://github.com/plotly/dash-renderer.

not to mention other potential JS UI component libraries.

It is possible to render almost any of these UI components inside a React component.

I chose react to be the basis for this implementation of the front-end library because the community is huge (there are so many good components out there) and because its declarative components and lifecycle management are a natural fit for Dash's declarative, reactive design. See https://medium.com/@plotlygraphs/introducing-dash-5ecf7191b503 for more.

I noticed that your APIs show images by setting the layout and the GraphWidget accepts on_click events. Can it be used in a similar way as matplotlib on jupyter and expose equivalent event information?

See https://plot.ly/dash/interactive-graphing for documentation on the available events of the dcc.Graph component. In particular, see the clickData property.

So the app must capture the mouse click coordinates and feed the corresponding image patch to the model.

The clickData property only works for clicks on points, it doesn't currently register clicks on arbitrary coordinates. However, I believe that @jimmybow made a component that would allow for arbitrary clicks in https://github.com/jimmybow/mydcc.

This could be used in tandem with a background image: https://plot.ly/python/images/

I would also be OK with adding this to the dcc.Graph component (source code: https://github.com/plotly/dash-core-components/blob/25e8e83f0195152a9e769b128ccc21674179a2e6/src/components/Graph.react.js) - a property like click_target='points | paper' or something. Here is a thread on how to do this in plotly.js: https://community.plot.ly/t/get-mouses-position-on-click/4145?source_topic_id=6410.

If you are interested in implementing this then please create a new issue in https://github.com/plotly/dash-core-components and we can work through the design together 🙂

are there some component widgets already available for those use cases such as image filtering, selecting a range of audio wave or video timeline to play and so on?

There are not but #178 (comment) addressed this. It would be very great to have an image processing dash component toolkit. In particular, it seems like there are some nice components that have already been written by the community:

• https://github.com/nitin42/react-imgpro
• https://coldiary.github.io/react-image-mapper/
• https://github.com/ethanselzer/react-cursor-position

Porting these components to Dash (described in https://plot.ly/dash/plugins) isn't that much work if you are familiar with JavaScript.
Plotly can also be contracted to build these components for you, see https://plot.ly/products/consulting-and-oem/

[chriddyp]

Continuing the discussion in the community forum here: https://community.plot.ly/t/interactive-image-callbacks/8110/1

[chriddyp]

video timeline to play

For this particular issue, a third party component has been written by the community here: https://community.plot.ly/t/modifying-a-dom-property-in-html-video-dash-video-component/7649