There seems to be an offset on some parts of the image while others are fine. I guess it has something to do with how jupyter lays out the objects and how the mouse position is recorded
To get consistent zooming behavior on images you need to set the image.sizing property to stretch. This will resize the image to exactly fit into the sizex by sizey area of the initial plot. Here are a few more tips that might be helpful for your use case.
Set image.sizing to 'stretch' (as stated above)
Set all of the layout margins of the figure to zero. This way the layout width/height will exactly match the resulting image width height. Otherwise youβll get some aspect ratio distortion.
Add a scatter trace with two points, positioned on opposite corners of the image, with opacity 0 so theyβre not visible. This will give the autosize logic something to work with, so the Autoscale button will work to reset the zoom.
Set layout.yaxis.scaleanchor to 'x'. This will constrain the zoom tool to preserve the aspect ratio, which I expect would be desirable when looking at medical imaging