Very slow restyle on hover

I have a stacked bar chart with a lot of traces, and I want to change the color of the “closest” trace on hover to highlight it.
I have implemented this with the plotly hover/unhover events and Plotly.restyle, and it works.

The problem is that it is very slow: up to 2 seconds to do the restyle, and it seems caused by the large amount of traces even though most of are not on display due to zooming.
If I disable my events, the hoverinfo displays snappily without a problem, so there should be a way to do this efficiently.

Is there a way for me to do the color change without calling restyle? Or should we try to optimize restyle in the first place?

There must be a way to optimize performance here. Could you share a code snippet of what you have attempted so far to help us help you?