I will follow up on Emil’s reply. I solved very similiar problem where my data was too big (it loaded for half an hour) to use first option. So I wrapped my whole data loading and preprocessing pipeline into a function that creates dataframe which is saved into global variable. Then I created BackgroundScheduler from apscheduler.schedulers.background which fires once a day after the data is ready and rewrites the global variables with data.
Of course one have to be really cautious with this approach. Changing global variables will be always kinda risky approach. You have to be careful to use references on your global variable and not copies. It will also depend how you use the data in layout creation - it can happen that the data changes when the user is browsing the dashboard.