Why would a background callback via Celery/Redis prevent initialization of actual app on Heroku?

Dash Python
1

Hi all,
building on top of my previous question:

I have an app on Heroku that has one user-triggered compute intensive task that gets sent to a background task worker via Celery/Redis.
If my understanding is correct, this should prevent overloading the web server, i.e. the app should always remain responsive. If there happen to be more users, they would have to wait longer for the results to come back, but the app should remain accessible.
However, what I observe, when load testing the system, is that after two or three jobs have been submitted to the queue the next time I try to access the app, I am served the landing page but it doesn’t initialize until the jobs are done (the tab title says “Updating…”). Even more confusing to me is that if I initialize the app several times (i.e. go to the URL and have the app wait for input), then start two or three jobs, I can still use the other previously initialized instances without any issues.

I tried upgrading the Heroku instances from the free tier to the paid for tier and increasing the number of workers for the app and the queue to two without success.

Tbh, I don’t even know how to go about troubleshooting this, so any input would be very welcome.

The basic code structure looks as follows:

if 'REDIS_URL' in os.environ:
    # Use Redis & Celery if REDIS_URL set as an env variable
    from celery import Celery
    celery_app = Celery(__name__, broker=os.environ['REDIS_URL'], backend=os.environ['REDIS_URL'])
    background_callback_manager = CeleryManager(celery_app)

else:
    # Diskcache for non-production apps when developing locally
    import diskcache
    cache = diskcache.Cache("./cache")
    background_callback_manager = DiskcacheManager(cache)


external_stylesheets = [dbc.themes.BOOTSTRAP, dbc.icons.FONT_AWESOME, dbc.icons.BOOTSTRAP]
app = Dash(__name__,
            external_stylesheets=external_stylesheets,
            background_callback_manager=background_callback_manager,
            )

server = app.server

The callback looks as follows, also following the docs. I have a progress bar and I’m returning the progress explicitly for the user:

@app.callback(
                    output = Output('growth_data_auto', 'data'),
                    inputs = Input('auto_fit_button', 'n_clicks'),
                    state = list_of_states,
                    background = True,
                    running = [
                                (Output('progress_count', 'style'),
                                        {'visibility': 'visible', 'justify-content': 'center'},
                                        {'visibility': 'hidden'},
                                ),
                                (Output("progress_bar", "style"),
                                        style_progress_bar,
                                        {"visibility": "hidden"},
                                ),
                                ),
                                
                                ],
                    progress = [Output('progress_bar', 'value'), Output('progress_bar', 'max'), Output('progress_count', 'children')],
                    cancel = [Input('cancel_auto_button', 'n_clicks')],
                    prevent_initial_callback = True
    )

def function_of_interest(set_progress, x, y, z):
     for i in range(n):
          set_progress(i, i, i)
          do_compute_intesive_task(x, y, z)

The procfile looks like this:

web: gunicorn app:server
queue: celery -A app.celery_app worker --concurrency=2 --loglevel=INFO

Thanks,
M

2

Hello @mretier,

If all workers are busy with a task, then there is no one to cater your small request.

You’ll have to check on Heroku and how many workers the app is actually spinning up.

3

@jinnyzor thanks for the comment, but that should never happen for the web app worker in the first place since it doesn’t do any heavy lifting (and without the background task many more than 2-3 instances can run, also why would the app be responsive still after initialization), right?

4

Depends on your setup, and how your app loads.

Does your first request go into a background callback?

5

That’s correct, I did not express myself clearly on this. The compute-intensive task is user-triggered upon button press once the app is initialized and data has been loaded.

6

In your code, it should be prevent_initial_call instead of prevent_initial_callback.

Your layout is automatically triggering the callback. See if that helps it.

1 Like
7

It was exactly that. The callback was called inadvertently because I put in the wrong parameter…

Thank you very much!

1 Like