Yeah I understand the hurdle @nikok . I am dealing with the same issue and I described my solution here: Automated Dashboard Python Dash - #5 by martin2097
I understand that updating global variables is not optimal but I did not come with any better solution (I work even with bigger data few gigabites big). I use scheduler for around a year in our internal setup with about 10-20 user and for last year there was no problem with it.
I will be glad if somebody will come with better and architecturally better solution 