Sharing data generated during background callback

Dash Python
1

Hi folks,

In the app I’m currently working on, I want to load and process data via a callback triggered by a file upload. The data at the end is saved in a global variable, which allows me to access the processed data throughout the app.

To avoid blocking the main thread due to the time-consuming processing (during the upload), I tried to move this logic into a background callback.

However, I noticed that anything created inside the background callback seems to be running in a different thread, and once the callback is finished, the data is essentially “lost”.

In my minimal example, the update_progress callback stores data in a global variable called DATASTORAGE. Another callback, get_odds, is supposed to read that variable and return only the odd numbers. But in that second callback, DATASTORAGE is empty.

I found a workaround using dcc.Store to persist the data across callbacks, which works fine.

Still, I’m wondering:

Is there any other way to retain data from a background callback?
Can I somehow share data across threads in Dash?
Thanks in advance!

from dash import Dash, DiskcacheManager, CeleryManager, html, Input, Output, dcc
import os
import dash_bootstrap_components as dbc
import time

DATASTORAGE = []


if 'REDIS_URL' in os.environ:
    # Use Redis & Celery if REDIS_URL set as an env variable
    from celery import Celery
    celery_app = Celery(__name__, broker=os.environ['REDIS_URL'], backend=os.environ['REDIS_URL'])
    background_callback_manager = CeleryManager(celery_app)

else:
    # Diskcache for non-production apps when developing locally
    import diskcache
    cache = diskcache.Cache("./cache")
    background_callback_manager = DiskcacheManager(cache)

modal = dbc.Modal([
        dbc.ModalHeader(
            dbc.ModalTitle("Progess..."),
            close_button=False
        ),
        dbc.ModalBody(children=[
            html.Progress(
                id="progress_bar",
                value="0",
                style={'width': '100%'}
            )
        ]),
        dbc.ModalFooter(
            dbc.Button(
                "Cancel",
                id="cancel_button",
                className="ms-auto",
                n_clicks=0
            )
        )
    ],
    id="modal",
    is_open=False,
    backdrop="static",
    keyboard=False
)

def create_layout(app):
    @app.callback(
        output=Output("out", "children"),
        inputs=[Input("btn-start", 'n_clicks')],
        background=True,
        running=[
            (Output('modal', 'is_open'), True, False)
        ],
        progress=[
            Output("progress_bar", "value"),
            Output("progress_bar", "max"),
            Output("workingfile", "children")
        ],
        cancel=Input("cancel_button", "n_clicks"),
        prevent_initial_call=True
    )
    def update_progress(set_progress, clicks):
        
        values = list(range(0, 100))
        
        total = 500

        val = 1
        set_progress((str(val), str(total), html.P("")))
            
        for value in values:
            val += total / (len(values) + 1)
            set_progress((str(val), str(total), html.P(f"{value}")))
            time.sleep(0.01)
            DATASTORAGE.append(value)

        set_progress((str(total), str(total), html.P("Done")))
        set_progress((str(0), str(total), html.P()))

        print("Datastorage populated in background callback")
        print(DATASTORAGE)

        return html.P(f"{DATASTORAGE}")
    
    @app.callback(
        Output("odds", "children"),
        Input("out", "children")
    )
    def get_odds(_):
        print("Second callback")
        odds = [x for x in DATASTORAGE if x %2 != 0]

        return html.P(f"{odds}")
    

    return html.Div(children=[
        modal,
        html.Button("Start process..", id="btn-start"),
        html.Div(children=[], id="out"),
        html.Div(children=[], id="odds"),
    ], 
    style={"padding" : "15px"}
    )




def main() -> None:
    app = Dash(
        prevent_initial_callbacks=True, 
        suppress_callback_exceptions=True,
        background_callback_manager=background_callback_manager
    )
    app.layout = create_layout(app)
    app.run(debug=True, port=2222)

if __name__ == "__main__":
    main()
2

Hi @Bakira

I am not really sure if that works, but you could try using the threading Lock to mutate the global variable.

import threading

class GlobalStorage:
    
    def __init__(self, data = []):
        self._data = data
        self.lock = threading.Lock()

    @property
    def data(self):
        return self._data
    
    @data.setter
    def data(self, update):
        with self.lock:
            self._data = update

    def append_data(self, value):
        with self.lock:
            self._data.append(value)

storage = GlobalStorage()

That said, web apps should have a centralised store for mutable state. The moment you leave the local environment, you cant make sure that a request that depends on a previous mutated local state gets routed to the same machine that mutated the state. Also, your current approach with the dcc.Store works as long as the state accounts for that specific user, the moment you have to also share the state between users you are more or less forced to use some kind of Database.

Hope this helps

3

Ah one more thing, it seams that Dash currently spawns a process rather then a thread. [BUG] Long callbacks create processes · Issue #3177 · plotly/dash · GitHub
Maybe this helps then multiprocessing.shared_memory — Shared memory for direct access across processes — Python 3.13.2 documentation