Long_callback fails with celery/redis

Summary

I’ve been developing a dash app that uses a long_callback, and for development I’ve been using a diskcache backend for my long_callback_manager, as recommended by the guide I found here: Long Callbacks | Dash for Python Documentation | Plotly

When I tried running my app using gunicorn, it failed to start because of something apparently wrong with diskcache. As such, I decided to switch to a celery/redis backend since that’s recommended for production anyway.

I got a redis server running (responds properly to redis-cli ping with PONG), and then started the app again. This time it started up fine, and all of the normal callbacks work, but the long_callback does not work.

Details:

  • The page more or less hangs, with the page title flashing between the normal title and the Updating... title,
    indicating that the app thinks it’s “waiting” for a response/update from the long_callback.
  • The values set by the running argument of the long_callback are set to their starting value,
    indicating that the app recognizes that the long_callback ought to run.
  • By placing a print statement as the first line within the long_callback function and seeing that
    it does not print, I’ve determined that the function never starts.
  • The failure happens both with gunicorn and without gunicorn.

These details all point to the problem being the celery/redis backend.
No errors are shown, neither on the client/browser nor on the server’s stdout/sterr.

How do I get a celery/redis backend working?

Code

app.py

import dash
import dash_bootstrap_components as dbc

from website.layout_main import define_callbacks, layout
from website.long_callback_manager import LONG_CALLBACK_MANAGER


app = dash.Dash(
    __name__,
    update_title="Loading...",
    external_stylesheets=[
        dbc.themes.BOOTSTRAP,
        "https://codepen.io/chriddyp/pen/bWLwgP.css"
    ],
    long_callback_manager=LONG_CALLBACK_MANAGER
)

app.title = "CS 236 | Project Submissions"
app.layout = layout
define_callbacks(app)
server = app.server  # expose for gunicorn

if __name__ == "__main__":
    app.run_server(debug=True, host="0.0.0.0")

website/long_callback_manager.py with diskcache (functional)

import os
import shutil

import diskcache
from dash.long_callback import DiskcacheLongCallbackManager

from util import RUN_DIR


cache_dir = os.path.join(RUN_DIR, "callback_cache")
shutil.rmtree(cache_dir, ignore_errors=True)  # ok if it didn't exist

cache = diskcache.Cache(cache_dir)

LONG_CALLBACK_MANAGER = DiskcacheLongCallbackManager(cache)

website/long_callback_manager.py with celery/redis (not functional)

from dash.long_callback import CeleryLongCallbackManager
from celery import Celery

celery_app = Celery(
    __name__,
    broker="redis://localhost:6379/0",
    backend="redis://localhost:6379/1"
)

LONG_CALLBACK_MANAGER = CeleryLongCallbackManager(celery_app)

website/layout_main.py

from typing import Union

import dash
import dash_bootstrap_components as dbc
from dash import dcc, html
from dash.dependencies import Input, Output, State

from util.authenticator import authenticate
from website import ID_LOGIN_STORE, NET_ID, PASSWORD
from website.tabs.config import define_config_callbacks, layout as config_layout
from website.tabs.log import define_log_callbacks, layout as log_layout
from website.tabs.submit import define_submit_callbacks, layout as submit_layout
from website.util import AUTH_FAILED_MESSAGE, STYLE_RED


# cache
LOGIN_INFO_EMPTY = {NET_ID: None, PASSWORD: None}
# button display modes
VISIBLE = "inline-block"
HIDDEN = "none"

# header
ID_LOGIN_BUTTON = "login-button"
ID_LOGGED_IN_AS = "logged-in-as"
ID_LOGOUT_BUTTON = "logout-button"
# tabs
ID_TAB_SELECTOR = "tab-selector"
ID_SUBMIT_TAB = "submit-tab"
ID_LOG_TAB = "log-tab"
ID_CONFIG_TAB = "config-tab"
# login modal
ID_LOGIN_MODAL = "login-modal"
ID_LOGIN_MODAL_NET_ID = "login-modal-net-id"
ID_LOGIN_MODAL_PASSWORD = "login-modal-password"
ID_LOGIN_MODAL_MESSAGE = "login-modal-message"
ID_LOGIN_MODAL_CANCEL = "login-modal-cancel"
ID_LOGIN_MODAL_ACCEPT = "login-modal-accept"
# logout modal
ID_LOGOUT_MODAL = "logout-modal"
ID_LOGOUT_MODAL_CANCEL = "logout-modal-cancel"
ID_LOGOUT_MODAL_ACCEPT = "logout-modal-accept"


layout = html.Div([
    dcc.Store(id=ID_LOGIN_STORE, storage_type="session", data=LOGIN_INFO_EMPTY),
    html.Div(
        [
            html.H2("BYU CS 236 - Project Submission Website", style={"marginLeft": "10px"}),
            html.Div(
                [
                    html.Div(id=ID_LOGGED_IN_AS, style={"display": HIDDEN, "marginRight": "10px"}),
                    html.Button("Log in", id=ID_LOGIN_BUTTON, style={"display": VISIBLE}),
                    html.Button("Log out", id=ID_LOGOUT_BUTTON, style={"display": HIDDEN})
                ],
                style={
                    "marginRight": "25px",
                    "display": "flex",
                    "alignItems": "center"
                }
            )
        ],
        style={
            "height": "100px",
            "marginLeft": "10px",
            "marginRight": "10px",
            "display": "flex",
            "alignItems": "center",
            "justifyContent": "space-between"
        }
    ),
    dcc.Tabs(id=ID_TAB_SELECTOR, value=ID_SUBMIT_TAB, children=[
        dcc.Tab(submit_layout, label="New Submission", value=ID_SUBMIT_TAB),
        dcc.Tab(log_layout, label="Submission Logs", value=ID_LOG_TAB),
        dcc.Tab(config_layout, label="View Configuration", value=ID_CONFIG_TAB)
    ]),
    dbc.Modal(
        [
            dbc.ModalHeader("Log In"),
            dbc.ModalBody([
                html.Div(
                    [
                        html.Label("BYU Net ID:", style={"marginRight": "10px"}),
                        dcc.Input(
                            id=ID_LOGIN_MODAL_NET_ID,
                            type="text",
                            autoComplete="username",
                            value="",
                            style={"marginRight": "30px"}
                        )
                    ],
                    style={
                        "marginBottom": "5px",
                        "display": "flex",
                        "alignItems": "center",
                        "justifyContent": "flex-end"
                    }
                ),
                html.Div(
                    [
                        html.Label("Submission Password:", style={"marginRight": "10px"}),
                        dcc.Input(
                            id=ID_LOGIN_MODAL_PASSWORD,
                            type="password",
                            autoComplete="current-password",
                            value="",
                            style={"marginRight": "30px"}
                        )
                    ],
                    style={
                        "display": "flex",
                        "alignItems": "center",
                        "justifyContent": "flex-end"
                    }
                ),
                html.Div(id=ID_LOGIN_MODAL_MESSAGE, style={"textAlign": "center", "marginTop": "10px"})
            ]),
            dbc.ModalFooter([
                html.Button("Cancel", id=ID_LOGIN_MODAL_CANCEL),
                html.Button("Log In", id=ID_LOGIN_MODAL_ACCEPT)
            ])
        ],
        id=ID_LOGIN_MODAL,
        is_open=False
    ),
    dbc.Modal(
        [
            dbc.ModalHeader("Log Out"),
            dbc.ModalBody("Are you sure you want to log out?"),
            dbc.ModalFooter([
                html.Button("Stay Logged In", id=ID_LOGOUT_MODAL_CANCEL),
                html.Button("Log Out", id=ID_LOGOUT_MODAL_ACCEPT)
            ])
        ],
        id=ID_LOGOUT_MODAL,
        is_open=False
    )
])


def on_click_login_modal_accept(net_id: Union[str, None], password: Union[str, None]) -> Union[str, None]:
    # validate
    if net_id is None or net_id == "":
        return "BYU Net ID is required."
    if password is None or password == "":
        return "Submission Password is required."
    # authenticate
    auth_success = authenticate(net_id, password)
    if auth_success:
        return None
    else:
        return AUTH_FAILED_MESSAGE


def define_callbacks(app: dash.Dash):
    @app.callback(Output(ID_LOGIN_MODAL, "is_open"),
                  Output(ID_LOGIN_MODAL_MESSAGE, "children"),
                  Output(ID_LOGOUT_MODAL, "is_open"),
                  Output(ID_LOGIN_STORE, "data"),
                  Input(ID_LOGIN_BUTTON, "n_clicks"),
                  Input(ID_LOGIN_MODAL_CANCEL, "n_clicks"),
                  Input(ID_LOGIN_MODAL_ACCEPT, "n_clicks"),
                  Input(ID_LOGOUT_BUTTON, "n_clicks"),
                  Input(ID_LOGOUT_MODAL_CANCEL, "n_clicks"),
                  Input(ID_LOGOUT_MODAL_ACCEPT, "n_clicks"),
                  State(ID_LOGIN_MODAL_NET_ID, "value"),
                  State(ID_LOGIN_MODAL_PASSWORD, "value"),
                  prevent_initial_call=True)
    def on_login_logout_clicked(
            n_login_clicks: int,
            n_login_cancel_clicks: int,
            n_login_accept_clicks: int,
            n_logout_clicks: int,
            n_logout_cancel_clicks: int,
            n_logout_accept_clicks: int,
            net_id: str,
            password: str):
        ctx = dash.callback_context
        btn_id = ctx.triggered[0]["prop_id"].split(".")[0]
        if btn_id == ID_LOGIN_BUTTON:
            # show the login modal (with no message)
            return True, None, dash.no_update, dash.no_update
        elif btn_id == ID_LOGIN_MODAL_CANCEL:
            # hide the login modal
            return False, dash.no_update, dash.no_update, dash.no_update
        elif btn_id == ID_LOGIN_MODAL_ACCEPT:
            # try to actually log in
            error_message = on_click_login_modal_accept(net_id, password)
            if error_message is None:  # login success!
                # hide the modal, update the login store
                return False, dash.no_update, dash.no_update, {NET_ID: net_id, PASSWORD: password}
            else:  # login failed
                # show the message and keep the modal open
                return dash.no_update, html.Span(error_message, style=STYLE_RED), dash.no_update, dash.no_update
        elif btn_id == ID_LOGOUT_BUTTON:
            # show the logout modal
            return dash.no_update, dash.no_update, True, dash.no_update
        elif btn_id == ID_LOGOUT_MODAL_CANCEL:
            # hide the logout modal
            return dash.no_update, dash.no_update, False, dash.no_update
        elif btn_id == ID_LOGOUT_MODAL_ACCEPT:
            # hide the logout modal and clear the login store
            return dash.no_update, dash.no_update, False, LOGIN_INFO_EMPTY
        else:  # error
            print(f"unknown button id: {btn_id}")  # TODO: better logging
            return [dash.no_update] * 4  # one for each Output

    @app.callback(Output(ID_LOGIN_BUTTON, "style"),
                  Output(ID_LOGGED_IN_AS, "children"),
                  Output(ID_LOGGED_IN_AS, "style"),
                  Output(ID_LOGOUT_BUTTON, "style"),
                  Input(ID_LOGIN_STORE, "data"),
                  State(ID_LOGIN_BUTTON, "style"),
                  State(ID_LOGGED_IN_AS, "style"),
                  State(ID_LOGOUT_BUTTON, "style"))
    def on_login_data_changed(login_store, login_style, logged_in_as_style, logout_style):
        # just in case no style is provided
        if login_style is None:
            login_style = dict()
        if logged_in_as_style is None:
            logged_in_as_style = dict()
        if logout_style is None:
            logout_style = dict()
        # are they logged in or not?
        if login_store[NET_ID] is None or login_store[PASSWORD] is None:
            # not logged in
            login_style["display"] = VISIBLE
            logged_in_as_style["display"] = HIDDEN
            logout_style["display"] = HIDDEN
            return login_style, None, logged_in_as_style, logout_style
        else:  # yes logged in
            login_style["display"] = HIDDEN
            logged_in_as_style["display"] = VISIBLE
            logout_style["display"] = VISIBLE
            return login_style, f"Logged in as '{login_store[NET_ID]}'", logged_in_as_style, logout_style

    # define callbacks for all of the tabs
    define_submit_callbacks(app)
    define_log_callbacks(app)
    define_config_callbacks(app)

website/tabs/submit.py

import os
import time
from io import StringIO
from typing import Callable, Dict, Union

import dash
import dash_bootstrap_components as dbc
import dash_gif_component as gif
from dash import dcc, html
from dash.dependencies import Input, Output, State
from dash.exceptions import PreventUpdate

from config.loaded_config import CONFIG
from driver.passoff_driver import PassoffDriver
from util.authenticator import authenticate
from website import ID_LOGIN_STORE, NET_ID, PASSWORD
from website.util import AUTH_FAILED_MESSAGE, save_to_submit, STYLE_DIV_VISIBLE, STYLE_DIV_VISIBLE_TOP_MARGIN, STYLE_HIDDEN, text_html_colorizer


# submit tab IDs
ID_SUBMISSION_ROOT_DIV = "submission-root-div"
ID_SUBMIT_PROJECT_NUMBER_RADIO = "submit-project-number-radio"
ID_UPLOAD_BUTTON = "upload-button"
ID_UPLOAD_CONTENTS = "upload-contents"
ID_FILE_NAME_DISPLAY = "file-name-display"
ID_SUBMISSION_SUBMIT_BUTTON = "submission-submit-button"
ID_SUBMISSION_OUTPUT = "submission-output"
ID_SUBMISSION_LOADING = "submission-loading"
# clear/refresh to submit again
ID_SUBMISSION_REFRESH_BUTTON = "submission-refresh-button"
ID_SUBMISSION_REFRESH_DIV = "submission-refresh-div"
ID_SUBMISSION_RESETTING_STORE = "submission-resetting-store"
# info modal
ID_SUBMISSION_INFO_MODAL = "submission-info-modal"
ID_SUBMISSION_INFO_MODAL_MESSAGE = "submission-info-modal-message"
ID_SUBMISSION_INFO_MODAL_ACCEPT = "submission-info-modal-accept"
# submission confirmation modal
ID_SUBMISSION_CONFIRMATION_MODAL = "submission-confirmation-modal"
ID_SUBMISSION_CONFIRMATION_MODAL_CANCEL = "submission-confirmation-modal-cancel"
ID_SUBMISSION_CONFIRMATION_MODAL_ACCEPT = "submission-confirmation-modal-accept"
# store to trigger submission
ID_SUBMISSION_TRIGGER_STORE = "submission-trigger-store"


LAYOUT_DEFAULT_CONTENTS = [
    html.H3("Upload New Submission"),
    html.P("Which project are you submitting?"),
    dcc.RadioItems(
        id=ID_SUBMIT_PROJECT_NUMBER_RADIO,
        options=[{
            "label": f" Project {proj_num}",
            "value": proj_num
        } for proj_num in range(1, CONFIG.n_projects + 1)]
    ),
    html.Br(),
    html.P("Upload your .zip file here:"),
    html.Div(
        [
            dcc.Upload(
                html.Button("Select File", id=ID_UPLOAD_BUTTON),
                id=ID_UPLOAD_CONTENTS,
                multiple=False
            ),
            html.Pre("No File Selected", id=ID_FILE_NAME_DISPLAY, style={"marginLeft": "10px"})
        ],
        style={
            "display": "flex",
            "justifyContent": "flex-start",
            "alignItems": "center"
        }
    ),
    html.Button("Submit", id=ID_SUBMISSION_SUBMIT_BUTTON, style={"marginTop": "20px"}),
    html.Div(id=ID_SUBMISSION_OUTPUT, style=STYLE_HIDDEN),
    html.Div(
        html.Div(
            gif.GifPlayer(
                gif=os.path.join("assets", "loading.gif"),
                still=os.path.join("assets", "loading.png"),
                alt="loading symbol",
                autoplay=True
            ),
            style={"zoom": "0.2"}
        ),
        id=ID_SUBMISSION_LOADING,
        style=STYLE_HIDDEN
    ),
    html.Div(
        [
            html.P("Reset the page to submit again:"),
            html.Button("Reset", id=ID_SUBMISSION_REFRESH_BUTTON),
        ],
        id=ID_SUBMISSION_REFRESH_DIV,
        style=STYLE_HIDDEN
    ),
    dbc.Modal(
        [
            dbc.ModalHeader("Try Again"),
            dbc.ModalBody(id=ID_SUBMISSION_INFO_MODAL_MESSAGE),
            dbc.ModalFooter([
                html.Button("OK", id=ID_SUBMISSION_INFO_MODAL_ACCEPT)
            ])
        ],
        id=ID_SUBMISSION_INFO_MODAL,
        is_open=False
    ),
    dbc.Modal(
        [
            dbc.ModalHeader("Confirm Submission"),
            dbc.ModalBody("Are you sure you want to officially submit?"),
            dbc.ModalFooter([
                html.Button("Cancel", id=ID_SUBMISSION_CONFIRMATION_MODAL_CANCEL),
                html.Button("Submit", id=ID_SUBMISSION_CONFIRMATION_MODAL_ACCEPT)
            ])
        ],
        id=ID_SUBMISSION_CONFIRMATION_MODAL,
        is_open=False
    )
]

layout = html.Div(
    [
        html.Div(LAYOUT_DEFAULT_CONTENTS, id=ID_SUBMISSION_ROOT_DIV),
        # having this store outside of the layout that gets reset means the long callback is not triggered
        dcc.Store(id=ID_SUBMISSION_TRIGGER_STORE, storage_type="memory", data=False)  # data value just flips to trigger the long callback
    ],
    style={
        "margin": "10px",
        "padding": "10px",
        "borderStyle": "double"
    }
)


def on_submit_button_clicked(
        proj_number: Union[int, None],
        file_name: Union[str, None],
        file_contents: Union[str, None],
        login_store: Union[Dict[str, str], None]) -> Union[str, None]:
    # validate
    if login_store is None or NET_ID not in login_store or PASSWORD not in login_store:
        return "There was a problem with the login store!"
    net_id = login_store[NET_ID]
    password = login_store[PASSWORD]
    if net_id is None or net_id == "" or password is None or password == "":
        return "You must log in before submitting."
    if proj_number is None:
        return "The project number must be selected."
    if not (1 <= proj_number <= CONFIG.n_projects):
        return "Invalid project selected."
    if file_name is None or file_name == "" or file_contents is None or file_contents == "":
        return "A zip file must be uploaded to submit."
    if not file_name.endswith(".zip"):
        return "The uploaded file must be a .zip file."
    # all good, it seems; return no error message
    return None


def run_submission(proj_number: int, file_contents: str, login_store: Dict[str, str], set_progress: Callable):
    # authenticate
    print("authenticate")
    net_id = login_store[NET_ID]
    password = login_store[PASSWORD]
    auth_success = authenticate(net_id, password)
    if not auth_success:
        set_progress([AUTH_FAILED_MESSAGE])
        return
    # write their zip file to the submit directory
    print("save_to_submit")
    save_to_submit(proj_number, net_id, file_contents)
    # actually submit
    print("actually submit")
    this_stdout = StringIO()
    this_stderr = StringIO()
    driver = PassoffDriver(net_id, proj_number, use_user_input=False, stdout=this_stdout, stderr=this_stderr)
    driver.start()  # runs in a thread of this same process
    while True:  # make sure we print the output at least once, even it it finishes super fast
        time.sleep(1)  # check output regularly
        # TODO: change saved final_result in PassoffDriver to diff/error info?
        # show results to the user
        output = list()
        output.append(html.P(f"submission for Net ID '{net_id}', project {proj_number}"))
        stdout_val = this_stdout.getvalue()
        output.append(html.Pre(text_html_colorizer(stdout_val)))
        # output.append(html.Br())
        stderr_val = this_stderr.getvalue()
        if stderr_val != "":
            output.append(html.P("ERRORS:"))
            output.append(html.Pre(text_html_colorizer(stderr_val)))
        set_progress([output])
        if not driver.is_alive():  # once it finishes, we're done too
            print("driver finished")
            # TODO: tack on diff info?
            if CONFIG.expose_test_cases and driver.final_result is not None and driver.final_result.has_failure_details:
                output.append(html.P(html.B("Use the \"Submission Logs\" tab to get more detailed information."), style={"marginTop": "20px"}))
                set_progress([output])
            break


def define_submit_callbacks(app: dash.Dash):
    @app.callback(Output(ID_FILE_NAME_DISPLAY, "children"),
                  Input(ID_UPLOAD_CONTENTS, "filename"),
                  prevent_initial_call=True)
    def on_select_file(filename: str):
        if filename is None or filename == "":
            return "No File Selected"
        return filename

    @app.callback(Output(ID_SUBMISSION_CONFIRMATION_MODAL, "is_open"),
                  Output(ID_SUBMISSION_INFO_MODAL, "is_open"),
                  Output(ID_SUBMISSION_INFO_MODAL_MESSAGE, "children"),
                  Output(ID_SUBMISSION_TRIGGER_STORE, "data"),
                  Input(ID_SUBMISSION_SUBMIT_BUTTON, "n_clicks"),
                  Input(ID_SUBMISSION_CONFIRMATION_MODAL_CANCEL, "n_clicks"),
                  Input(ID_SUBMISSION_CONFIRMATION_MODAL_ACCEPT, "n_clicks"),
                  Input(ID_SUBMISSION_INFO_MODAL_ACCEPT, "n_clicks"),
                  State(ID_SUBMIT_PROJECT_NUMBER_RADIO, "value"),
                  State(ID_UPLOAD_CONTENTS, "filename"),
                  State(ID_UPLOAD_CONTENTS, "contents"),
                  State(ID_LOGIN_STORE, "data"),
                  State(ID_SUBMISSION_TRIGGER_STORE, "data"),
                  prevent_initial_call=True)
    def on_submission_submit_clicked(
            n_submit_clicks: int,
            n_confirmation_cancel_clicks: int,
            n_confirmation_accept_clicks: int,
            n_info_accept_clicks: int,
            proj_number: int,
            file_name: str,
            file_contents: str,
            login_store: Dict[str, Union[str, None]],
            submission_trigger_store: bool):
        ctx = dash.callback_context
        trigger_id = ctx.triggered[0]["prop_id"].split(".")[0]
        if trigger_id == ID_SUBMISSION_SUBMIT_BUTTON:
            if n_submit_clicks is None:
                raise PreventUpdate
            # validate
            error_message = on_submit_button_clicked(proj_number, file_name, file_contents, login_store)
            if error_message is None:  # good to go
                # show the confirmation modal
                return True, dash.no_update, dash.no_update, dash.no_update
            else:
                # show the error message in the info modal
                return dash.no_update, True, error_message, dash.no_update
        elif trigger_id == ID_SUBMISSION_CONFIRMATION_MODAL_CANCEL:
            if n_confirmation_cancel_clicks is None:
                raise PreventUpdate
            # hide the confirmation modal
            return False, dash.no_update, dash.no_update, dash.no_update
        elif trigger_id == ID_SUBMISSION_CONFIRMATION_MODAL_ACCEPT:
            if n_confirmation_accept_clicks is None:
                raise PreventUpdate
            # hide the confirmation modal and trigger on_submit_confirmed
            return False, dash.no_update, dash.no_update, not submission_trigger_store  # just flip the value, whatever it is
        elif trigger_id == ID_SUBMISSION_INFO_MODAL_ACCEPT:
            if n_info_accept_clicks is None:
                raise PreventUpdate
            # hide the info modal
            return dash.no_update, False, dash.no_update, dash.no_update
        else:  # error
            print(f"unknown button id: {trigger_id}")  # TODO: better logging
            return [dash.no_update] * 4  # one for each Output

    @app.long_callback(
        progress=[Output(ID_SUBMISSION_OUTPUT, "children")],
        progress_default=[dash.no_update],  # I'll set stuff manually, thank you very much
        output=[Output(ID_SUBMISSION_REFRESH_DIV, "style")],
        inputs=[
            Input(ID_SUBMISSION_TRIGGER_STORE, "data"),
            State(ID_SUBMIT_PROJECT_NUMBER_RADIO, "value"),
            State(ID_UPLOAD_CONTENTS, "contents"),
            State(ID_LOGIN_STORE, "data")
        ],
        running=[  # hide the submit button when it starts running, then keep it hidden so they have to refresh to submit again
            (Output(ID_SUBMISSION_SUBMIT_BUTTON, "style"), STYLE_HIDDEN, STYLE_HIDDEN),
            (Output(ID_SUBMISSION_LOADING, "style"), STYLE_DIV_VISIBLE, STYLE_HIDDEN),
            (Output(ID_SUBMISSION_OUTPUT, "style"), STYLE_DIV_VISIBLE_TOP_MARGIN, STYLE_DIV_VISIBLE_TOP_MARGIN)
        ],
        prevent_initial_call=True)
    def on_submit_confirmed(
            set_progress: Callable,
            submission_trigger_store: bool,
            proj_number: int,
            file_contents,
            login_store):
        print("start of long callback")
        set_progress([None])
        print("after update progress to None")
        # actually run the submission
        run_submission(proj_number, file_contents, login_store, set_progress)
        print("finished run_submission")
        # wait a sec to make sure all `set_progress` calls can finish
        time.sleep(1)
        # show the "clear page" message/button
        print("about to return")
        return [STYLE_DIV_VISIBLE_TOP_MARGIN]

    @app.callback(Output(ID_SUBMISSION_ROOT_DIV, "children"),
                  Input(ID_SUBMISSION_REFRESH_BUTTON, "n_clicks"),
                  prevent_initial_call=True)
    def on_submission_refresh_clicked(n_refresh_clicks: int):
        # reset everything
        return LAYOUT_DEFAULT_CONTENTS

Environment

python version

$ python --version
Python 3.9.6

installed packages

$ pip list
Package                   Version
------------------------- ---------
amqp                      5.0.6
billiard                  3.6.4.0
Brotli                    1.0.9
celery                    5.1.2
click                     7.1.2
click-didyoumean          0.0.3
click-plugins             1.1.1
click-repl                0.2.0
dash                      2.0.0
dash-bootstrap-components 0.13.1
dash-core-components      2.0.0
dash-gif-component        1.1.0
dash-html-components      2.0.0
dash-table                5.0.0
dill                      0.3.4
diskcache                 5.2.1
Flask                     2.0.1
Flask-Compress            1.10.1
greenlet                  1.1.1
gunicorn                  20.1.0
itsdangerous              2.0.1
Jinja2                    3.0.1
kombu                     5.1.0
MarkupSafe                2.0.1
multiprocess              0.70.12.2
orjson                    3.6.3
pip                       21.2.4
plotly                    5.3.1
prompt-toolkit            3.0.20
psutil                    5.8.0
pytz                      2021.1
redis                     3.5.3
setuptools                56.0.0
six                       1.16.0
SQLAlchemy                1.4.23
tenacity                  8.0.1
vine                      5.0.0
wcwidth                   0.2.5
Werkzeug                  2.0.1
wheel                     0.37.0

redis

$ redis-server --version
Redis server v=6.2.5 sha=00000000:0 malloc=jemalloc-5.1.0 bits=64 build=2a367e4b809d24de
$ redis-cli ping
PONG
$ curl localhost:6379
curl: (52) Empty reply from server
$ sudo ss -lptn 'sport = :6379'
State         Recv-Q        Send-Q               Local Address:Port               Peer Address:Port       Process
LISTEN        0             511                      127.0.0.1:6379                    0.0.0.0:*           users:(("redis-server",pid=373,fd=6))

OS details

Windows, WSL 2, Ubuntu

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Hi @rkechols ,

Did you manage to get it work?

I’m just doing my first steps with @long_callback (I’m only focusing on the Celery/Redis backend). I managed to get the documentation example work but now I’m struggling to add long_callbacks to a multi-page app…

Eventually, I’ll also need to get everything working under gunicorn.

I have not gotten it to work. Would you mind posting your code for comparison?

I posted my solution for the documentation example. link

Once I get it work also for my multi-page app, I’ll post it to the thread.

Independently starting the celery process was indeed what was needed. Thanks so much for the help!

To the dash/plotly people: please update your tutorial/documentation to make this clear.