Detect where user clicked on image

Dash Python
1

Hi,
I’m looking for a way to detect coordinates where user clicked on image.

The idea is that user sees screenshot of desktop machine refreshed every second and has ability to click somewhere on it and expect reaction as if he actually clicked with mouse attached to streamed desktop machine.

I’m aware how to program backend on server side, but I’m stuck with frontend in dash.

Right now I managed to print screenshot every second with html.Img.
I create component with current screenshot with:

def get_screen_grab():
    return html.Img(
        src='data:image/jpeg;base64,{}'.format(system.get_screen()),
        style={
            'width': '75%',
            'height': '75%'
        })

Function is triggered every second by dcc.Interval component

html.Div([
        html.Div(id='live-screen'),
        dcc.Interval(
            id='screen-interval-component', interval=1000, n_intervals=0)])

Interval triggers callback that puts html.Img to live-screen Div.

@app.callback(
    Output('live-screen', 'children'),
    [Input('screen-interval-component', 'n_intervals')])
def grab_screen(n):
    return get_screen_grab()

How can I get coordinates where user clicked on streamed image? (It may be some other component - it doesn’t have to be html.Img). What I need is information that user clicked point that’s for example in 60% of desktop screen height and 30% of desktop screen width.

I appreciate any help with that. :slight_smile:

2

It might be possible to do this in a dcc.Graph component. You could put the image in the background and then remove all the lines and ticks as in this example and then add a scatter trace containing a dense grid of transparent points on top of it that you’ll use to determine the x and y coordinates of the click.

2 Likes
3

Doing it in a Graph component works well to leverage the re_layout events and the graph drawing capabilities.
However, in my use case I need to update the background image every 1 second.
It is not practical to re-create the Graph component every 1 second.

Do you have an idea to just update the image without re-creating the Graph component?

Or another idea of how to get the mouse click coordinates for an image displayed with the html.Img component?

Thanks a lot!

4

Another option would be to use Dash Leaflet,

http://dash-leaflet.herokuapp.com/

I haven’t tried it out, so I am not sure how performance would be as compared to using a Graph component, but I guess the following would work,

  • Fix the bounds of the map (to disable pan/zoom)
  • Load the image using an ImageLayer, and update this layer as desired (maybe you would need an extra layer for caching to avoid flickering)
  • Capture clicks using the map click events

You might also want to use simple CRS to avoid the (lat, lon) mapping.

5

Thanks for the reply! Dash Leaflet and the possibility to have a caching layer sounds great.
That makes me think there might be a way to do caching with a dcc.Store. but I’m not dash-expert enough to imagine how that would work between the client and server side.

I also found that when using the Graph component, there is the fig.update_layout_images(source=...) method that can update the background image in the Figure to avoid re-creating it every second.
However, after more experimenting, it seems that the overhead of (re)creating a go.Figure is small, the main part seems to be serializing the image data itself.
For reducing that, base64 encoding seems to help. Something like this yields visible improvement over just passing the file path string.

encoded_image = base64.b64encode(open(path, 'rb').read())
fig.update_layout_images(source=f'data:image/png;base64,{encoded_image.decode()}')

In any case lowering the resolution of the image on the server side improves refresh speed the most.

Hope these pointers help somebody!

For me the best solution I found so far is the mouse and keyboard event of ipywidgets: ipyevents
I wish there was the same generic container component for Dash!