Incremental Data Load Progress Bar

Hey Dash folk,

I’m trying to work out a way of showing the progress of a data load & process that takes a long time. I’m processing air quality data from a number of different measurement locations around the globe and calling this data from a database, processing it and then plan on visualising it for the user.
However each of these calls for one measurement site takes approximately one second, which is fine on its own but I’m looking at comparing around 80 sites and the time taken is almost linear. Not only is this boring for the user to wait with no indication of what is going on but also causes the server to break if there is no response from the database within 30 seconds.
My plan was to load the data bit by bit and show the user a progress indicator (maybe something like the graduated bar component or just something like ‘13/80 sites processed’).
Is there a way of doing this in Dash? Something like a callback that kicks out the progress but still keeps going until all the data is processed? I’ve tried some callbacks that kick off other callbacks but get stuck in endless loops.
Any help much appreciated!

p.s. It may be the case that I just need a faster database but due to budget and resource limits I’m trying other methods!

You probably want to run the long running thing as a background task using a task queue like Celery or Redis Queue. I made a simple example of something similar, should give you an idea of one way of accomplishing this. I’ve seen a few other examples around too.

1 Like

This does look like the solution I was after! Thanks!

1 Like