Loading and filtering large dataset in Multipage app

Dash Python
1

Hello dash community,
I have been developing a dashboard with Dash for a few months now and also read the forum here. Most of the time I was able to solve my problems with the answers here in the community.
However, now I have a problem which I can’t fix.

I have a multipage app. I load a text file into a dataframe with more than 610k rows. The file is bigger than 1,2 GB. It is also possible that the file is still growing, or a larger file needs to be loaded in another project.

The file is needed in different pages.
I never display all rows of the file. Always summarized information are displayed. But you have the possibility to filter in the file. That means the big table is filtered in a callback function and then summarized and returned. In most cases a maximum of 30 rows are displayed.

Now I have the problem that I the file has become very large and it comes to a memory error. Dash Dashboard cannot be loaded.

How to deal with large files? How should I load and filter the file so that dashboard runs without any problem.

Thanks a lot.

2

How are you storing the data at the moment? Are the source data static for each application?

3

The file is stored in a variable.I also tried to work with cache, but I didn’t understand exactly how the cahcing works. But also there I got an exception. The data is the same on every page. The data is the same on each page but is analyzed differently.

4

Could vaex be a solution?