Understanding background callbacks via Celery/Redis better

Dash Python
1

Hi all,
I am in the process of developing a Dash app which is hosted on Heroku and contains one compute intensive task.
This task is user-triggered upon button click and spins off a task via Celery/Redis as explained in the docs.
I am struggling to really understand what is going on at the Redis end or with the communication between web and worker.

My celery implementation code follows the Dash documentation. On the Heroku end, I’m using the Redis addon.

if 'REDIS_URL' in os.environ:
    # Use Redis & Celery if REDIS_URL set as an env variable
    from celery import Celery
    celery_app = Celery(__name__, broker=os.environ['REDIS_URL'], backend=os.environ['REDIS_URL'])
    background_callback_manager = CeleryManager(celery_app)

else:
    # Diskcache for non-production apps when developing locally
    import diskcache
    cache = diskcache.Cache("./cache")
    background_callback_manager = DiskcacheManager(cache)


external_stylesheets = [dbc.themes.BOOTSTRAP, dbc.icons.FONT_AWESOME, dbc.icons.BOOTSTRAP]
app = Dash(__name__,
            external_stylesheets=external_stylesheets,
            background_callback_manager=background_callback_manager,
            )

server = app.server

The callback looks as follows, also following the docs. I have a progress bar and I’m printing the progress explicitly for the user:

@app.callback(
                    output = Output('growth_data_auto', 'data'),
                    inputs = Input('auto_fit_button', 'n_clicks'),
                    state = list_of_states,
                    background = True,
                    running = [
                                (Output('progress_count', 'style'),
                                        {'visibility': 'visible', 'justify-content': 'center'},
                                        {'visibility': 'hidden'},
                                ),
                                (Output("progress_bar", "style"),
                                        style_progress_bar,
                                        {"visibility": "hidden"},
                                ),
                                ),
                                
                                ],
                    progress = [Output('progress_bar', 'value'), Output('progress_bar', 'max'), Output('progress_count', 'children')],
                    cancel = [Input('cancel_auto_button', 'n_clicks')],
                    prevent_initial_callback = True
    )

def function_of_interest(set_progress, x, y, z):
     for i in range(n):
          set_progress(i, i, i)

My understanding is that every user of the app would create a connection to Redis once they click on the button.
I.e. after starting the app (heroku restart), I would expect 0 connections to Redis. However, the number of clients is 1 directly after restarting and a few seconds later there are about 10 clients (this is without anyone accessing the Dash app).

Another problem: As outlined in the code, I’m using the progress functionality of the background callback. This appears to be spawning many more connections to Redis (shouldn’t it be able to keep on using the first connection or close the old one?) The problem is that the free Redis implementation in Heroku only allows for only 20 clients (I’m assuming this is equivalent to connections, but not sure) and subsequently kills my computation. This does not occur when running the same callback without the progress functionality.

I hope someone can shed some light on my confusion.

Thanks a lot in advance.
-M

2

Read this discussion. May be this can help you here https://community.plotly.com/t/long-callback-with-celery-redis-how-to-get-the-example-app-work-uk49s. If yes then please mentioned me here.

3

@uk49s thank you for the suggestion. Unfortunately, the problem in the linked thread is not quite the same. My background callback does work and the Job queuing appears to be working, too.
The whole process just generates too many connections to Redis and it is unclear why it would generate so many.

4

I think I found a solution to my problem even though I don’t think I really understand what is going on.

When setting the BROKER_POOL_LIMIT=0 in Celery (as suggested here) connection pooling is disabled and connections will be established and closed for every use (according to the Celery docs), i.e. the expected behavior from my perspective.

According to the Stackoverflow discussion if the limit is set to its default value (BROKER_POOL_LIMIT=10), each connection spawned by Celery in the pool may still create several connections to Redis which in turn results in exceeding the connection limit.

If someone has a concise writeup of what is going on, I’d still appreciate it.