Hi, I am trying to figure out how to speed up getting data from my sql server when loading my dash app. I currently have a dash app with a couple tabs, where one of the tabs (let’s say tab A), contains dropdowns and shows relevant charts when the user selects an option in the dropdown. When dash initially runs, it retrieves all data I need from the sql server using pyodhc.
The issue is that it takes a couple minutes for this process as there are more than a hundred different queries to the database, puts them respectively into a pandas dataframe, does some very simple calculations, and saves the dfs as global variables (for them to be then called in callbacks). The querying and the dataframe calculations are done in a different file than from where the dash app runs with callbacks. Once dash is fully loaded, going through dropdowns is very fast and the charts are shown immediately when an option is selected.
Is there a solution to optimize the initial load of the data?
Here are some ideas that I’ve thought of:
(1) Only having the most important/prioritized data load first, then having the rest load when user selects the relevant dropdown option. Would this be able to work? If the user selects tab B, then comes back to tab A, I don’t want the data to load again.
(2) After querying, putting it in a pandas dataframe, and doing calculations, I tried converting to json to see if it would help load the page faster. It didn’t make a difference.
(3) I was thinking of saving the data I would need into txt file then having the dash app read off of that upon loading, but not sure of its efficiency.
(4) I looked into utilizing redis and celery but not sure if this is a simple solution.