# @long\_callback with Celery & Redis - how to get the example app work?

**URL:** <https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663>\
**Category:** Dash Python\
**Created:** [October 18, 2021, 10:36am UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663 "2021-10-18T10:36:55Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![sislvacl](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@sislvacl](https://community.plotly.com/u/sislvacl)\
**Post date:** [October 18, 2021, 10:36am UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/1 "2021-10-18T10:36:55Z")

</div>

Hello,

today, I am trying to update my app to Dash 2.0 mainly to take the advantage of the @long\_callback decorator.

I have troubles getting the [Example with Celery/Redis app](https://dash.plotly.com/long-callbacks) from Dash documentation to work.

I run Redis in docker using the following command:

`$ docker run -p 6379:6379 redis`

My dash app in module called `playground.py` and its content is copy-pasted from the documentation:

```auto
import time
import dash
from dash import html
from dash.long_callback import CeleryLongCallbackManager
from dash.dependencies import Input, Output
from celery import Celery

celery_app = Celery(
    __name__ , broker="redis://localhost:6379/0", backend="redis://localhost:6379/1"
)
long_callback_manager = CeleryLongCallbackManager(celery_app)

app = dash.Dash( __name__ , long_callback_manager=long_callback_manager)

app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
    ],
    cancel=[Input("cancel_button_id", "n_clicks")],
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]

if __name__ == " __main__":
    app.run_server(debug=True)

```

I run the app by executing:

`$ python playground.py`

The app starts and I get **no errors**. I open the app in the browser to see this:

 ![image](https://us1.discourse-cdn.com/flex024/uploads/plot/original/3X/f/5/f541792fdf959a18ecb2f2639d59e3786e08a59c.png)

My assumption is that the `callback()` is triggered on page load and that’s why the ‘Run Job!’ button is disabled. The button never gets activated so I assume that the job never finishes. The app does not throw any errors.

What am I missing? Perhaps I need another separate process for the Celery worker? I tried to follow the [Celery tutorial](https://docs.celeryproject.org/en/stable/getting-started/first-steps-with-celery.html#first-steps) but couldn’t adapt it to use with Dash. Thanks for any advice!

---

<div class="post-metadata">

**Author:** ![amarv](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/amarv/32/3407_2.png) [@amarv](https://community.plotly.com/u/amarv)\
**Post date:** [October 18, 2021, 1:53pm UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/2 "2021-10-18T13:53:00Z")

</div>

Same issue for me.

---

<div class="post-metadata">

**Author:** ![sislvacl](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@sislvacl](https://community.plotly.com/u/sislvacl)\
**Post date:** [October 18, 2021, 1:57pm UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/3 "2021-10-18T13:57:50Z")

</div>

As I suspected, a separate process for the Celery worker is needed. In fact, two processes - one for the worker and another one for something called _beat_.

How to run the example app:

## 1) Start the redis server

Either with docker:

`$ docker run -p 6379:6379 redis`

Or without docker:

`$ redis-server`

## 2) Start Celery beat and worker processes

(two separate processes in separate terminal windows):

`$ celery -A playground.celery_app beat --loglevel=INFO`

and

`$ celery -A playground.celery_app worker --loglevel=INFO`

## 3) Start the Dash app

`$ python playground.py`

## edit:

The _beat_ process from point 2) is not necessary as @amarv pointed out.

---

<div class="post-metadata">

**Author:** ![amarv](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/amarv/32/3407_2.png) [@amarv](https://community.plotly.com/u/amarv)\
**Post date:** [October 18, 2021, 2:32pm UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/4 "2021-10-18T14:32:06Z")

</div>

Great solution! For me, my Dash app is defined in `application.py` and long callbacks started working by also running `celery -A application.celery_app worker`. I didn’t need to run Celery beat.

---

<div class="post-metadata">

**Author:** ![sislvacl](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@sislvacl](https://community.plotly.com/u/sislvacl)\
**Post date:** [October 18, 2021, 2:38pm UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/5 "2021-10-18T14:38:23Z")

</div>

You are right. It works also without the beat process. I’ll need to do some more reading about the purpose of the beat process.

Anyways, I’m happy it works for you. I’m now struggling with making long\_callbacks work in a multi-page app. Have you tried that?

---

<div class="post-metadata">

**Author:** ![amarv](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/amarv/32/3407_2.png) [@amarv](https://community.plotly.com/u/amarv)\
**Post date:** [October 18, 2021, 2:43pm UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/6 "2021-10-18T14:43:08Z")

</div>

Nice! Sorry, I haven’t tried it with multi-page Dash apps. Maybe you’ll have issues with circular imports that is common with Dash multi-page apps, so you’ll have to use the suggested approach of separating into two Python files the app declaration and server declaration (see [URL Routing and Multiple Apps | Dash for Python Documentation | Plotly](https://dash.plotly.com/urls) section “Structuring a Multi-Page App”).

---

<div class="post-metadata">

**Author:** ![rkechols](https://avatars.discourse-cdn.com/v4/letter/r/898d66/32.png) [@rkechols](https://community.plotly.com/u/rkechols)\
**Post date:** [October 19, 2021, 1:59am UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/7 "2021-10-19T01:59:40Z")

</div>

So to make sure I understand the solution, even though the `playground.py` file creates a `Celery` object that is hooked into the dash app which is being run, the celery app also needs to be started on its own?

---

<div class="post-metadata">

**Author:** ![sislvacl](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@sislvacl](https://community.plotly.com/u/sislvacl)\
**Post date:** [October 19, 2021, 9:58am UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/8 "2021-10-19T09:58:38Z")

</div>

I am not sure about how exactly it all works nor am I sure about the correct terminology but I’ll try to formulate it the way I understand it…

The Celery app is created in `playground.py` as an object of type `Celery`. The Celery app on its own does not run the asynchronous tasks. It probably only sends the information about tasks to be executed to Redis and also keeps an eye on finished tasks which are also collected in Redis.

There must be a separate worker process whose job is to look for tasks submitted to Redis, execute the tasks (asynchronously from the main dash process), and once finished, submit the results to Redis. This is the process that is started with the command:

`$ celery -A playground.celery_app worker --loglevel=INFO`

Please anyone correct me if I’m wrong somewhere…

---

<div class="post-metadata">

**Author:** ![sislvacl](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@sislvacl](https://community.plotly.com/u/sislvacl)\
**Post date:** [October 19, 2021, 2:22pm UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/9 "2021-10-19T14:22:32Z")

</div>

## Enabling @long\_callback for a multi-page app

Making the @long\_callback work in a multi-page app is quite simple. Assuming a project structure as follows:

```auto
- app.py
- index.py
- apps
   |-- __init__.py
   |-- app1.py
   |-- app2.py

```

In `app.py`:

```auto
import dash
from dash.long_callback import CeleryLongCallbackManager
from celery import Celery

celery_app = Celery(
    __name__ , broker="redis://localhost:6379/0", backend="redis://localhost:6379/1",
    include=['apps.app1', 'apps.app2']
)

long_callback_manager = CeleryLongCallbackManager(celery_app)

app = Dash( __name__ , long_callback_manager=long_callback_manager)

server = app.server

```

Note the **`include`** parameter of the Celery object - here, you have to set paths to all python modules in which you define functions decorated with` @long_callback`.

For the rest of the app, see the [instructions](https://dash.plotly.com/urls) in official documentation (Structuring a Multi-Page app).

The celery worker will be started by executing:  
`$ celery -A app.celery_app worker --loglevel=INFO`

---

<div class="post-metadata">

**Author:** ![amarv](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/amarv/32/3407_2.png) [@amarv](https://community.plotly.com/u/amarv)\
**Post date:** [October 19, 2021, 6:52pm UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/10 "2021-10-19T18:52:49Z")

</div>

> [@sislvacl](#):
>
> `include=['apps.app1', 'apps.app2']`

I found that I could avoid this need, and have the Celery worker automatically detect all `long_callback` by including in `index.py` a line:

```python
from app import celery_app # noqa: F401

```

and then running the Celery worker with

```bash
celery -A index.celery_app worker --loglevel=INFO

```

---

<div class="post-metadata">

**Author:** ![sislvacl](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@sislvacl](https://community.plotly.com/u/sislvacl)\
**Post date:** [October 21, 2021, 9:31am UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/11 "2021-10-21T09:31:56Z")

</div>

I just hit the wall with `@long_callback` realising that it currently does NOT support pattern-matching dependencies.

I created a [feature request](https://github.com/plotly/dash/issues/1813) for this functionality. You can add your comment there if you also find this feature important.

---

<div class="post-metadata">

**Author:** ![ajm](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@ajm](https://community.plotly.com/u/ajm)\
**Post date:** [October 28, 2021, 7:18am UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/12 "2021-10-28T07:18:48Z")

</div>

As a workaround, you can use a normal callback with pattern matching and store the inputs as a JSON string in a hidden div.  
Then add a long\_callback with the hidden div as input.  
Not pretty but works for me.

---

<div class="post-metadata">

**Author:** ![sislvacl](https://avatars.discourse-cdn.com/v4/letter/s/ecccb3/32.png) [@sislvacl](https://community.plotly.com/u/sislvacl)\
**Post date:** [October 28, 2021, 8:06am UTC](https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work/57663/13 "2021-10-28T08:06:26Z")

</div>

That’s a smart solution! I wonder if something like this could be implemented behind the scenes to enable pattern-matching dependencies for @long\_callback out of the box? @jmmease