I think the payload is just too big for dcc.Store. Even with just one input and two outputs, load time is around 1 full minute.
Rows are 250k. [ specific error: Failed to execute 'setItem' on 'Storage': Setting the value of 'mdstore' exceeded the quota. ]
Does anyone have any links on storing data locally in cache? As mentioned here.
Any help appreciated. Thanks!
The space available in store of type session is either 5MB or 10MB. This depends on the browser and is not because of the dcc.Store object. If you change the type to memory, you can store many more MB, i have been able to save files of 180MB.
Just be aware that load times will increase as data needs to be transfered between server and browser each time the callback executes. Secondly, when running your server on a cloud provider, additional constraints may apply. For example AWS API gateway imposes a 6MB limit per HTTP request, i.e. per incoming and outgoing request individually.
In the final example above, i didn’t store the dataframe in the browser, I just stored the filter settings. This is based on the assumption that the source data is the same for every user. Is this assumption no longer true in your app?
Do you know how long it takes to apply the filter to the entire dataframe? If this is fairly fast (or can be made fairly fast) it may not be necessary to cache local data - you could instead go for the approach I suggested above - find a way of performing the filter on the server very quickly, and then shift towards an approach of performing the filter serverside whenever it’s needed.