Slow performance when dealing with large Datasets

[seventy77]

Hello,

I’m quite new to using dash library and in my experience, I have found a limitation when I’m working with a large amount of data because I’m using dash components dcc.store in order to share data between callbacks, so atm I’m getting into a point where I see that all the work that I have done to build my dashboard is not enough because the performance of the page is poor and week.
Is there an alternative to mitigate this? I’m working with 5 million rows and this is not the biggest number, this could variate depending on the user input.

I also use a VM that has a lot of computational power but also is taking a while to update the other dash components.

dcc .store is the most effective way to share data between callbacks?

I think that’s the main issue of my code. I need to find a way to share data quick because the postprocessing part is very expensive in time so I need to do it only once and then shared the result to dcc.store

[jinnyzor]

Hello @seventy77,

You should look into serverside outputs here:

Show and Tell - Server Side Caching

The Store component in Dash makes it easy to share state between callbacks. Under the hood, the data are stored as JSON in the browser. This approach is chosen to keep the server stateless (i guess), but it has a few drawbacks As the data are stored in JSON, you must convert objects from/to JSON in the beginning/end of each callback Since the callbacks are executed server side while the data are stored client side, the data will be sent across the wire every time a callback is invoked The maxi…

This keeps you from having to exchange it back and forth with the network.

[seventy77]

Hi Jinny this works with for local host? Because at this point in time is taking a while with my local host. Idk how long it would take once I shared though a link

[jinnyzor]

Take a look here, and see if any of this can help you.

Speed issue after saving with Serversideoutput

That depends on how you implemented the data cleanup. If you didn’t implement any data cleanup, data will accumulate indefinitly, with 244 MB for each callback invocation. So I would say that 11.6 GB is expected.

Also, if you are using inputs, especially typed, make sure you are using debounce