Option to toggle annotation visibility when clicking legend?

I am struggling to find a solution to my problem.
I have a graph, where I want the traces to be arrows connecting nodes, so I have read the best solution for this is to add annotations on the traces.
I have done this and it works great, but now I want to utilize legendgroups and be able to toggle traces visibility. It works for the traces, but annotations stay on no matter what. What I am looking for is kind of legendgroup parameter support for annotations, but it seems there is none. I have looked for solutions to this problem and didn’t find anything satisfactory.
I would love any help, hacky solutions too!