Cancelling outdated background callbacks when switching pages

Dash Python
1

I opened an issue about this here.

When I switch between pages, the incomplete background callbacks for visualizations of the previous page finish before the background callbacks for the next page are picked up by Celery workers.

I’m hoping to find a strategy to cancel background callbacks when I switch pages so that the delay that a user experiences is minimal, and so that the Celery workers don’t get bogged down by unnecessary work.

Dash implements a version of this with the ‘oldJob’ request payload, specifying jobs to cancel if the current page seems to be requesting the execution of the same background callback, but this isn’t available for page-level cancellation.

Any recommendations would be appreciated.

2

Hello @jkunstle,

You could tie in your background callbacks to cancel from a page navigation:

###background callback
cancel=[Input("_pages_location", "pathname")]
3 Likes
3

@jinnyzor I’m going to try that. That’d make a lot of sense. I disregarded this pattern initially because I didn’t think that I could guarantee that a the cancellation “signal” from the URL change would always precede the triggering of a new callback relevant to the current page. I’m going to try implementing this and make as ‘answered’ if this works.

4

From testing on my own stuff, the _pages_location prop is pretty immediate. If your other callbacks trigger from inside the other page’s layout, I dont think you will run into any issues.

5

@jinnyzor So you’d use _pages_location rather than a dcc.Location object?

6

When using pages, _pages_location is a default dcc.Location. It’s how dash knows what layout to cater.

Dash pages are just callbacks triggered from the _pages_location that then passes the children to the _pages_container. :slight_smile:

7

Ahhh okay that’s awesome. Trying that now.

1 Like
8

So… weird behavior. I can use:

cancel=[Input("_pages_location", "pathname"),],

On a single background callback but not more than one. I tried using it on all the callbacks on a page and no pages in my app rendered aside from the navbar, but when I only used the cancel argument on a single callback on that page everything worked.

Any ideas on this? It wouldn’t really make sense for the cancel Inputs to have to be unique across all callbacks, would it?

9

If running in debug, were there any error messages?

10

Oh yeah, I forgot about the web debugger.

In the callback for output(s):
  _pages_location.id
Output 0 (_pages_location.id) is already in use.
Any given output can only have one callback that sets it.
To resolve this situation, try combining these into
one callback function, distinguishing the trigger
by using `dash.callback_context` if necessary.

I’m confused because I’m not outputting to ‘_pages_location.id’ anywhere.

Both instances are in the format:

    cancel=[
        Input("_pages_location", "pathname"),
    ],
11

Just toss, allow_duplicate=True onto it and try it.

12

That’s an Output() level prop right? That’s the confusing bit- I’m not pointing to ‘_pages_location.id’ anywhere, I’m just taking ‘_pages_location.pathname’ as an input.

13

Right, just give it a shot, haha.

14

Lol ‘unexpected keyword argument’

This feels like a bug that ought to be reported to me- having an input act like an output doesn’t make any sense.

15

Can you give an MRE to work with? Obviously, with just a local disk cache.

16

Yes I’ll try- I’ll get back to this thread in a little while, maybe tomorrow.

17

Here, I got one that isnt working obviously due to the error:

import time
import os

import dash
from dash import Dash, DiskcacheManager, CeleryManager, Input, Output, html, callback, dcc

if 'REDIS_URL' in os.environ:
    # Use Redis & Celery if REDIS_URL set as an env variable
    from celery import Celery
    celery_app = Celery(__name__, broker=os.environ['REDIS_URL'], backend=os.environ['REDIS_URL'])
    background_callback_manager = CeleryManager(celery_app)

else:
    # Diskcache for non-production apps when developing locally
    import diskcache
    cache = diskcache.Cache("./cache")
    background_callback_manager = DiskcacheManager(cache)

app = Dash(__name__, background_callback_manager=background_callback_manager, use_pages=True, pages_folder='')

app.layout = html.Div(
    [
        dcc.Link('page1', '/'), dcc.Link('page2', '/2'),
        dash.page_container
    ]
)

layout1 = [html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!")]

layout2 = [html.Div([html.P(id="paragraph_id2", children=["Button not clicked"])]),
        html.Button(id="button_id2", children="Run Job!"),
        html.Button(id="cancel_button_id2", children="Cancel Running Job!")]

dash.register_page('page1', path='/', layout=layout1)
dash.register_page('page1', path='/2', layout=layout2)

@callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    background=True,
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
    ],
    cancel=[Input("cancel_button_id", "n_clicks"), Input('_pages_location', 'href')],
)

@callback(
    output=Output("paragraph_id2", "children"),
    inputs=Input("button_id2", "n_clicks"),
    background=True,
    running=[
        (Output("button_id2", "disabled"), True, False),
        (Output("cancel_button_id2", "disabled"), False, True),
    ],
    cancel=[Input("cancel_button_id2", "n_clicks"), Input('_pages_location', 'href')],
)
def update_clicks(n_clicks):
    time.sleep(4.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run(debug=True)
1 Like
18

Ok, so looking at how the cancel works, it creates a callback for those components. So, this just needs to be done as allow_duplicate=True somehow. :stuck_out_tongue:

19

Yeah I see what you mean. Wonder if that should be the default for those ‘Output’ callbacks, especially given that this seems like a reasonable performance-optimization design pattern.

(edit) also, your building an MRE for this in like, 10 minutes is amazing.

20

Haha, thanks I really just altered the background cancel callback example to use pages. :slight_smile: