Background callback with dash extensions and flask

Dash Python
1

I have a Dash app inside a Flask app and am using DashProxy from dash extensions. My problem is getting the background callback with Celery to work. Celery shows the following error: Received un registered task of type 'long_callback. The app is initialized like this:

from celery_app import celery_app
from dash_extensions.enrich import NoOutputTransform, DashProxy


background_callback_manager = CeleryManager(celery_app)


def init_dash(server):

    app = DashProxy(__name__, server=server, url_base_pathname='/dashboard/',
                    transforms=[NoOutputTransform()],
                    prevent_initial_callbacks=True,
                    background_callback_manager=background_callback_manager
                    )

    ### Define Layout
    
    init_callbacks(app)
    
    app.register_celery_tasks()
    
    return app


def init_callbacks(app):

    @app.callback(
        Output('out', 'children'),
        Input('in', 'n_clicks'),
        prevent_initial_call=True,
        background=True,
        manager=background_callback_manager
    )
    def process(in, out):

      # Some long-running calculation

      return out

In another file i have a function to stitch the Flask and Dash apps together:

from flask import Flask


def create_app():

    app = Flask(__name__)

    from .app_dash import init_dash
    app = init_dash(app)

    return app.server

And finally in the app is run in another file with this code:

from flask_wrapper import create_app


app = create_app()


if __name__ == '__main__':

    app.run(host='127.0.0.1', port=8080, debug=False)

Since there are some callbacks without an output I would like to keep using DashProxy. In order to get Celery to work do I maybe have to move app.register_celery_tasks() to a different location or change my import statements? Glad if anyone could help me with this issue :slight_smile:

2

Hi there! Have you tried adding
from dash.long_callback import CeleryLongCallbackManager

Best,
Eliza

3

Just tried it and unfortunately the same error appears. Is it possible that initializing Dash inside a function leads to this behavior?

4

I also cannot get background callbacks working when using DashProxy from dash-extensisons. In my case celery worker does not register the background callback.

5

I was also having this issue, and have possibly made some progress, although my current test does not check whether I maintain the dash_extensions functionality.

Anyway, here is a pytest that works using dash[testing] for the dash_duo fixture, and celery[pytest] for the celery_app and celery_worker fixtures:

def test_background_callback_with_dash_proxy2(dash_duo, celery_app, celery_worker):
    """
    Now check with the background callback using DashProxy
    """
    import time
    import dash
    from dash import CeleryManager
    from dash_extensions.enrich import DashProxy, Output, Input, html, dcc

    proxy_app = DashProxy(__name__)
    proxy_app.layout = html.Div(
        children=[
            html.Div(id="example-div", children=["some content"]),
            dcc.Input(id="example-input"),
        ],
    )

    @proxy_app.callback(
        Output("example-div", "children"),
        Input("example-input", "value"),
        background=True,
        prevent_initial_call=True,
    )
    def update_div(text):
        return text

    manager = CeleryManager(celery_app)
    dash_app = dash.Dash(__name__, background_callback_manager=manager)
    proxy_app.hijack(dash_app)

    # This must come after the hijack
    celery_worker.reload()  # So that worker can see the newly registered task

    dash_duo.start_server(dash_app)

    content = dash_duo.wait_for_element_by_css_selector("#example-div")
    assert content.text == "some content"

    dash_duo.find_element("#example-input").send_keys("new content")
    # For the next 20 seconds, keep checking whether the content has changed
    for _ in range(20):
        content = dash_duo.wait_for_element_by_css_selector("#example-div")
        if content.text == "new content":
            break
        time.sleep(1)

    content = dash_duo.wait_for_element_by_css_selector("#example-div")
    assert content.text == "new content"

Basically, hijack a regular dash.Dash app that has the background_callback_manager using the DashProxy app, then run the regular app.

I haven’t thoroughly checked, but I believe the hijack uses transforms when registering callbacks on the dash.Dash app.

This is not an ideal solution, but hopefully it is a step towards a better one.

6

I had the same issue, but I have a potential workaround where I can get dash_extensions, background_callback_manager, and flask to work together, but I’m not sure if it will work in more complex flask applications. Basically, instead of instantiating the dash app inside of flask with app = init_dash(app), I turned it around and instantiate the flask app inside of the dash app, and then grab the flask server for dash. In this way, I can still access both Flask and Dash routes.

In flask_app.py:

import flask
from werkzeug.serving import run_simple


def create_flask_app():
    app = flask.Flask(__name__)

    @app.route("/")
    def home():
        return "Hello, Flask!"

    return app


if __name__ == "__main__":
    run_simple("localhost", 8050, create_flask_app())

and in dash_app.py

import time

from celery import Celery
from dash import CeleryManager
from dash_extensions.enrich import (
    DashProxy,
    Input,
    Output,
    RedisBackend,
    ServersideOutputTransform,
    callback,
    html,
)

from flask_app import create_flask_app

# *** INITIALIZE CELERY ***

REDIS_URL = "redis://127.0.0.1:6379/0"
REDIS_HOST = "127.0.0.1"


celery_app = Celery(__name__, broker=REDIS_URL, backend=REDIS_URL)
background_callback_manager = CeleryManager(celery_app)

# *** INSTANTIATE FLASK APP, AND GRAB THE SERVER ***
server = create_flask_app()

URL_BASE = "/bkgd/"

app = DashProxy(
    __name__,
    server=server,
    routes_pathname_prefix=URL_BASE,
    background_callback_manager=background_callback_manager,
    transforms=[ServersideOutputTransform(backends=[RedisBackend(host=REDIS_HOST)])],
)

app.layout = html.Div(
    [
        html.Div(
            [
                html.P(id="paragraph_id", children=["Button not clicked"]),
                html.Progress(id="progress_bar", value="0"),
            ]
        ),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)


@callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    background=True,  # triggers dash to use background callbacks here
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
        (
            Output("paragraph_id", "style"),
            {"visibility": "hidden"},
            {"visibility": "visible"},
        ),
        (
            Output("progress_bar", "style"),
            {"visibility": "visible"},
            {"visibility": "hidden"},
        ),
    ],
    cancel=Input("cancel_button_id", "n_clicks"),
    progress=[Output("progress_bar", "value"), Output("progress_bar", "max")],
    prevent_initial_call=True,
)
def update_progress(set_progress, n_clicks):
    total = 5
    print("\nUpdating progress!!!")
    for i in range(total + 1):
        set_progress((str(i), str(total)))
        time.sleep(1)

    return f"Clicked {n_clicks} times"


app.register_celery_tasks()  # required for dash_extensions to use celery

if __name__ == "__main__":
    app.run(debug=True)

in requirements.txt

dash
dash_extensions
celery
redis

I have a redis server running on my system. I first run celery worker with:

celery -A dash_app:celery_app worker --loglevel=INFO --concurrency=2

and then launch the app with:

python dash_app.py

I’ve published the codebase here: GitHub - brentkendrick/dash_extensions_flask

Unfortunately, while most of the background callback functions work, the “cancel” option does not currently work when using dash_extensions DashProxy, as apparently others have also found: Dash-extensions, Serverside(), cancel of background callback. Not sure if @Emil is able to chime in on a potential fix, but in any case I love using the dash_extensions ServersideOutputTransform function for pushing large dataframes around in my callbacks and avoiding the json serialization.

I haven’t yet tested the workaround by @TimC, but will give it a try and see if it resolves the issue with the background callback “cancel” function.