Best Practice Question: Long-running data collection + processing task

Dash Python
,
1

Our app queries data from a database when the data becomes relevant to users. The data is cached server-side afterward in Redis. We’re using rq + Redis for our job queue needs.

Parallel database queries are executed in the workers and take ~30sec to complete, with a max of ~5 minutes.

Question:

While the workers are loading and transforming data for data to be stored in Redis (later to be aggregated and visualized) we want to show the user something informative, or at the very least something to indicate that work is being done.

We aren’t able to use background_callbacks for this purpose, but they have a nice persistent loading-state for the components that rely on those long callbacks.

We’re using dcc.Interval components to check whether the data required to created the requested visualizations is available in Redis but this causes a dcc.Loading component on our graphs to ‘flash’ which looks terrible.

Does anyone have any suggestions of most optimal design patterns to solve this problem? Would love any input.

2

Hello @jkunstle,

Instead of having the n_intervals tied to a callback that directly interacts with the graphs, you could use a chained callback:

@app.callback(
    Output('tempStore', 'data'),
    Input('refresh','n_intervals'),
    prevent_intial_call=True
)
def checkData(n1):
    if n1:
        if testingfunction():
            return time.time()
    raise PreventUpdate

@app.callback(
    Output('graphDiv','children'),
    Input('tempStore','data_timestamp'),
    prevent_initial_call=True
)
def updateCharts(ts):
    if ts:
        return updateCharts()
    raise PreventUpdate

Where tempStore is a dcc.Store which gets updated with the last time that the cache actually updated. Thus triggering an update to your charts.

1 Like
3

@jinnyzor That’s a pretty good solution, I like that.

Do you think there’s a more optimal way than dcc.Interval to accomplish this or is this what that component is for?

A follow-up also:

Is there a good way to manually set the loading state of the figures that are waiting for data to True? I would like the user to see a loading wheel or something while the data is being downloaded and processing to indicate that work is being done.

4

Yes and no. Yes - its for frequent updates, No - if you plan on only updating it once in a blue moon, it could cause unnecessary server queries.

Why can’t you use a background callback? This could update the store if possible, plus has the ability to set props during runtime.

Are your figures all in the same div and grouped together so that you could display something over it?

5

@jinnyzor

I’ve resisted using background callbacks because that’s another deployment component to manage. If we’re using rq+Redis for our data-store then having Celery also handling jobs concurrently feels hacky.

Maybe I need to get over this though. It’s worth a shot to try it and to see if it works well for us!

I really appreciate the help, this has set me on the right path.

I’m going to use background callbacks that wait for the data to become available in Redis, which will have the nice side-effect of removing burden from my server-workers and will also work nicely with Dash’s loading integration with background_callbacks.

1 Like