đź“Ł Dash Labs 0.3.0: @app.long_callback support

Hi all!

I wanted to highlight a new Dash 2 preview feature has landed in the version 0.3.0 of Dash Labs.

It is a new callback decorator called @app.long_callback. I’ve included the current documentation inline below.

As with other Dash Labs features, the API will not be final/stable until the feature is released as part of Dash 2, but we would really appreciate your feedback! Thanks!


Dash Labs introduces a new callback decorator called @long_callback. This decorator is designed to make it easier to create callback functions that take a long time to run, without locking up the Dash app or timing out.

@long_callback is designed to support multiple backend executors. Two backends are currently implemented:

  • A Flask-Caching backend that runs callback logic in a separate process and stores the results in a Flask-Caching cache. This is the easiest backend to use for local development.
  • A Celery backend that runs callback logic in a celery worker and returns results to the Dash app through a Celery broker like RabbitMQ or Redis.

The @long_callback decorator supports the same arguments as the normal @callback decorator, but also includes support for 3 additional arguments that will be discussion below: running, cancel, and progress.

Enabling long-callback support

In Dash Labs, the @long_callback decorator is enabled using the LongCallback plugin. To support multiple backends, the LongCallback plugin is, itself, configured with either a FlaskCachingCallbackManager or CeleryCallbackManager object. Furthermore, in addition to the LongCallback plugin, the FlexibleCallback and HiddenComponents plugins must be enabled as well. Here is an example of configuring an app to enable the @long_callback decorator using the Flask-Caching backend.

import dash
import dash_labs as dl

# ## FlaskCaching
from flask_caching import Cache
flask_cache = Cache(config={"CACHE_TYPE": "filesystem", "CACHE_DIR": "./cache"})
long_callback_manager = FlaskCachingCallbackManager(flask_cache)


app = dash.Dash(__name__, plugins=[
    dl.plugins.FlexibleCallbacks(),
    dl.plugins.HiddenComponents(),
    dl.plugins.LongCallback(long_callback_manager)
])

This configuration requires the Flask-Caching package which can be installed with:

$ pip install Flask-Caching

Example 1: Simple background callback

Here is a simple example of using the @long_callback decorator to register a callback function that updates an html.P element with the number of times that a button has been clicked. The callback uses time.sleep to simulate a long-running operation.

import time
import dash
import dash_html_components as html
import dash_labs as dl
from dash_labs.plugins import FlaskCachingCallbackManager

# ## FlaskCaching
from flask_caching import Cache
flask_cache = Cache(config={"CACHE_TYPE": "filesystem", "CACHE_DIR": "./cache"})
long_callback_manager = FlaskCachingCallbackManager(flask_cache)


app = dash.Dash(__name__, plugins=[
    dl.plugins.FlexibleCallbacks(),
    dl.plugins.HiddenComponents(),
    dl.plugins.LongCallback(long_callback_manager)
])

app.layout = html.Div([
    html.Div([
        html.P(id="paragraph_id", children=["Button not clicked"])
    ]),
    html.Button(id='button_id', children="Run Job!"),
])


@app.long_callback(
    output=dl.Output("paragraph_id", "children"),
    args=dl.Input("button_id", "n_clicks"),
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 2: Disable button while callback is running

In the previous example, there is no visual indication that the long callback was running. It is also possible to click the “Run Job!” button multiple times before the original job has the chance to complete. This example addresses these shortcomings by disabling the button while the callback is running, and re-enabling it when the callback completes.

This is accomplished using the running argument to @long_callback. This argument accepts a list of 3-element tuples. The first element of each tuple should be an Output dependency object referencing a property of a component in the app layout. The second elements is the values that the property should be set to while the callback is running, and the third element is the value the property should be set to when the callback completes.

This example uses running to set the disabled property of the button to True while the callback is running, and False when it completes.

import time
import dash
import dash_html_components as html
import dash_labs as dl
from dash_labs.plugins import FlaskCachingCallbackManager

# ## FlaskCaching
from flask_caching import Cache
flask_cache = Cache(config={"CACHE_TYPE": "filesystem", "CACHE_DIR": "./cache"})
long_callback_manager = FlaskCachingCallbackManager(flask_cache)

app = dash.Dash(__name__, plugins=[
    dl.plugins.FlexibleCallbacks(),
    dl.plugins.HiddenComponents(),
    dl.plugins.LongCallback(long_callback_manager)
])

app.layout = html.Div([
    html.Div([
        html.P(id="paragraph_id", children=["Button not clicked"])
    ]),
    html.Button(id='button_id', children="Run Job!"),
])

@app.long_callback(
    output=dl.Output("paragraph_id", "children"),
    args=dl.Input("button_id", "n_clicks"),
    running=[
        (dl.Output("button_id", "disabled"), True, False),
    ],
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 3: Cancelable callback

This example builds on the previous example, adding support for canceling a long-running callback using the cancel argument to the @long_callback decorator. The cancel argument should be set to a list of Input dependency objects that reference a property of a component in the app’s layout. When the value of this property changes while a callback is running, the callback is canceled. Note that the value of the property is not significant, any change in value will result in the cancellation of the running job (if any).

import time
import dash
import dash_html_components as html

import dash_labs as dl
from dash_labs.plugins import FlaskCachingCallbackManager

# ## FlaskCaching
from flask_caching import Cache
flask_cache = Cache(config={"CACHE_TYPE": "filesystem", "CACHE_DIR": "./cache"})
long_callback_manager = FlaskCachingCallbackManager(flask_cache)

app = dash.Dash(__name__, plugins=[
    dl.plugins.FlexibleCallbacks(),
    dl.plugins.HiddenComponents(),
    dl.plugins.LongCallback(long_callback_manager)
])

app.layout = html.Div([
    html.Div([
        html.P(id="paragraph_id", children=["Button not clicked"])
    ]),
    html.Button(id='button_id', children="Run Job!"),
    html.Button(id='cancel_button_id', children="Cancel Running Job!"),
])

@app.long_callback(
    output=dl.Output("paragraph_id", "children"),
    args=dl.Input("button_id", "n_clicks"),
    running=[
        (dl.Output("button_id", "disabled"), True, False),
        (dl.Output("cancel_button_id", "disabled"), False, True),
    ],
    cancel=[dl.Input("cancel_button_id", "n_clicks")],
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 4: Progress bar

This example uses the progress argument to the @long_callback decorator to update a progress bar while the callback is running. The progress argument should be set to an Output dependency object that references a tuple of two properties of a component in the app’s layout. The first property will be set to the current iteration of the task that the decorated function is executing, and the second property will be set to the total number of iterations.

When a dependency object is assigned to the progress argument of @long_callback, the decorated function will be called with a new special argument as the first argument to the function. This special argument, named set_progress in the example below, is a function handle that the decorated function should call in order to provide updates to the app on its current progress. The set_progress function accepts two positional argument, which correspond to the two properties specified in the Output dependency object passed to the progress argument of @long_callback. The first argument to set_progress should be the current iteration of the task that the decorated function is executing, and the second argument should be the total number of iterations.

import time
import dash
import dash_html_components as html
import dash_labs as dl
from dash_labs.plugins import FlaskCachingCallbackManager, CeleryCallbackManager

# ## FlaskCaching
from flask_caching import Cache
flask_cache = Cache(config={"CACHE_TYPE": "filesystem", "CACHE_DIR": "./cache"})
long_callback_manager = FlaskCachingCallbackManager(flask_cache)

# ## FlaskCaching
flask_cache = Cache(config={"CACHE_TYPE": "filesystem", "CACHE_DIR": "./cache"})
long_callback_manager = FlaskCachingCallbackManager(flask_cache)

app = dash.Dash(__name__, plugins=[
    dl.plugins.FlexibleCallbacks(),
    dl.plugins.HiddenComponents(),
    dl.plugins.LongCallback(long_callback_manager)
])

app.layout = html.Div([
    html.Div([
        html.P(id="paragraph_id", children=["Button not clicked"]),
        html.Progress(id="progress_bar"),
    ]),
    html.Button(id='button_id', children="Run Job!"),
    html.Button(id='cancel_button_id', children="Cancel Running Job!"),
])


@app.long_callback(
    output=dl.Output("paragraph_id", "children"),
    args=dl.Input("button_id", "n_clicks"),
    running=[
        (dl.Output("button_id", "disabled"), True, False),
        (dl.Output("cancel_button_id", "disabled"), False, True),
        (dl.Output("paragraph_id", "style"), {"visibility": "hidden"}, {"visibility": "visible"}),
        (dl.Output("progress_bar", "style"), {"visibility": "visible"}, {"visibility": "hidden"}),
    ],
    cancel=[dl.Input("cancel_button_id", "n_clicks")],
    progress=dl.Output("progress_bar", ("value", "max")),
)
def callback(set_progress, n_clicks):
    total = 10
    for i in range(total):
        time.sleep(0.5)
        set_progress(str(i + 1), str(total))
    # Set progress back to indeterminate state for next time
    set_progress(None, None)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Celery configuration

Here is an example of configuring the LongCallback plugin to use Celery as the execution backend rather than a background process with FlaskCaching.

import dash
import dash_labs as dl
from dash_labs.plugins import CeleryCallbackManager, FlaskCachingCallbackManager

## Celery on RabbitMQ
from celery import Celery
celery_app = Celery(__name__, backend='rpc://', broker='pyamqp://')
long_callback_manager = CeleryCallbackManager(celery_app)

app = dash.Dash(__name__, plugins=[
    dl.plugins.FlexibleCallbacks(),
    dl.plugins.HiddenComponents(),
    dl.plugins.LongCallback(long_callback_manager)
])

See the Celery documentation for more information on configuring a Celery app instance.

10 Likes

Wow, this is amazing! I can’t wait to try it out :smiley:

Amazing stuff!

I love the idea of being able to update a value whilst the function is executing like the progress does. It would be awesome to be able to update more than just a pair of (current iteration, total iterations). For instance I’d love to be able to write:

@app.long_callback(
    ...,
    progress=dict(
        progress1=dl.Output("component1", "attribute1"),
        progress2=dl.Output("component2", ("attribute1", "attribute2"))
    )
)
def my_function(set_progress, *args, **kwargs):
    # Do something
    set_progress({"progress1": ..., "progress2": ...})
    # Do some more stuff
    set_progress({"progress1": ..., "progress2": ...})
   # And some more
    return ...

Is that technically feasible or are there constraints to what can be update while the callback is running?

EDIT: the idea would be for instance to edit a notification message of where the long callback is at.

Thanks for taking a look @RenaudLN.

Yeah, that’s a really good idea. it should be possible to loosen the tuple restriction and basically just pass through whatever set_progress is called with and match it against the dependency grouping provided to the progress argument. I opened long_callback: Make `set_progress` arguments more flexible · Issue #27 · plotly/dash-labs · GitHub to track it.

-Jon

This looks great. Is there any plan to allow the object that the long callback returns remain on the server?
ie like Emil’s Server Side Cache:
show-and-tell-server-side-caching

At least for my use-case if I have a long-running callback its usually because I am processing a lot of data that i then want to access later (without having to perform the long process again). I usually don’t want to send all that data back to the browser, just filtered view of it.
Thanks!

Hi @jmmease,

Thanks for your information. It’s amazing! :smiley:

Could I ask two more questions:

  1. Does @app.long_callback support normal dash input/output/state? Like:
  1. Does normal dash input/output/state can use together with dl.Output/dl.Input? Like:

Thanks and best regards,

1 Like

I modified the long callback to accept any dependency mapping for the progress part, here’s an example of what it could look like. Could do a PR if that helps :slight_smile:

long_callback_arbitrary_output

7 Likes

Arbitrary dependency support is not available in version 0.4.0. See đź“Ł Dash Labs 0.4.0: @long_callback caching and Windows support.

1 Like

We really like the ServersideInput/ServersideOutput paradigm that @Emil came up with, and we would like to pull something along these lines into Dash eventually, but it probably won’t be for Dash 2.0.

If/when we get to that point, the idea is that long_callback could output intermediary results to ServersideOutput dependencies to combine the background computation and intermediary result caching features.
-Jon

Hi, I couldn’t find the answer anywhere so I did some experimenting and the following works:

@app.long_callback(
    output=dl.Output("paragraph_id", "children"),
    args=(
        dl.Input("button_id", "n_clicks"),
        dl.State("button_id", "n_clicks_timestamp"),
    )
)
def callback(n_clicks):
    print(n_clicks) # (2, 1627354204364)
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]

Note that the function has exactly 1 argument which is a tuple of callback args.

Hi,

I am experimenting the longcallback with Dash 2.0 and the function works great with Diskcache, however, it did not work with Celery. I tried the code with both local Windows and Azure VM, but I always got an error:

Windows: redis.exceptions.ConnectionError: Error 10061 connecting to localhost:6379. No connection could be made because the target machine actively refused it.

Linux: Connection refused

Any thoughts on it?

Are you running a Redis server on port 6379?

Yes, I got the Redis server ( https://github.com/MSOpenTech/redis/releases/download/win-3.2.100/Redis-x64-3.2.100.zip) as suggested in this post: redis - Error while starting Celery worker on django Error 10061 No connection could be made because the target machine actively refused it - Stack Overflow. However, the callback is still not responding.

Also, if the app is hosted in an Azure Web Service, how will the Redis server be hosted?

I just hit the wall with @long_callback realising that it does NOT support pattern-matching dependencies. This is clearly stated in the documentation. I just wished I read the docs properly before starting to code… :sweat_smile:

What is the reason why the @long_callback does not support pattern-matching dependencies out of the box? Is the support planned for the future versions of Dash?

I created a feature request for this.

1 Like

I tried both the examples mentioned in the documentation in Long Callbacks | Dash for Python Documentation | Plotly.

None of the two (DiskCache and Celery) seems to working on my system. It shows “Button not clicked” even after clicking number of times. Attached image:

I am using redis on docker container for Celery.

Hi @saurabh11baghel ,

have you seen this thread? @long_callback with Celery & Redis - how to get the example app work?

Hi @sislvacl , Thanks for providing the link. Got the idea.

Hi Emil,

I am fairly new to Dash, and this is my first message in the community. I hope it is alright to reply to your message here (I think it is relevant).

I have been using your awesome work on DashProxy (with MultiplexerTransform()) because I need callbacks that have same outputs. However, when I try to use long_callbacks with DashProxy in the example above (Example 4: Progress bar) I get the following error:

Traceback (most recent call last):
  File "/home/nima/.local/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/nima/.local/lib/python3.8/site-packages/flask/app.py", line 1511, in full_dispatch_request
    self.try_trigger_before_first_request_functions()
  File "/home/nima/.local/lib/python3.8/site-packages/flask/app.py", line 1563, in try_trigger_before_first_request_functions
    self.ensure_sync(func)()
  File "/home/nima/.local/lib/python3.8/site-packages/dash_extensions/enrich.py", line 223, in _setup_server
    self.blueprint.register_callbacks(super())
  File "/home/nima/.local/lib/python3.8/site-packages/dash_extensions/enrich.py", line 149, in register_callbacks
    o = cbp.outputs if len(cbp.outputs) > 1 else cbp.outputs[0]
IndexError: list index out of range

The only change I make is app = dash.Dash to app = DashProxy in Exmple 4. Is it possible to use long_callbacks in DashProxy? If not, is there any other way to have long callbacks in DashProxy?

I am using dash==2.4.0 and dash_extensions=0.1.1.

Thank you!
/Nima

@Neema Your observation is correct, long_callbacks are not support by dash-extensions :slight_smile: