Hello dash community,
I have been developing a dashboard with Dash for a few months now and also read the forum here. Most of the time I was able to solve my problems with the answers here in the community.
However, now I have a problem which I can’t fix.
I have a multipage app. I load a text file into a dataframe with more than 610k rows. The file is bigger than 1,2 GB. It is also possible that the file is still growing, or a larger file needs to be loaded in another project.
The file is needed in different pages.
I never display all rows of the file. Always summarized information are displayed. But you have the possibility to filter in the file. That means the big table is filtered in a callback function and then summarized and returned. In most cases a maximum of 30 rows are displayed.
Now I have the problem that I the file has become very large and it comes to a memory error. Dash Dashboard cannot be loaded.
How to deal with large files? How should I load and filter the file so that dashboard runs without any problem.
Thanks a lot.