Dash Background Callbacks on Render - how to set up celery background workers

[Pippo]

Hi guys,

I am trying to upgrade my app to use Background Callbacks, so that my app does not freeze if I have more users than web-workers. I understand that PythonAnywhere does not support Celery so will not support Background Callbacks, so I have first moved my Standard Callback Dash app to Render, this seems to be working following the very helpful guide here:

Migrating from Heroku: How to use Render to deploy a Python Dash app (Solution)

Hello World! Like many other people who are having to migrate from Heroku to other providers (), I found Render’s servers as a free hosting option. After brute-forcing a bunch, I found the setting to allow one to deploy a Python Dash app on Render. My test application has this skeleton: import dash import dash_auth import dash_bootstrap_components as dbc from dash import dcc, html, Output, Input, State, dash_table app = dash.Dash(__name__, meta_tags=[ …

I wondered if anyone has been able to successfully configure the Celery Background Worker in Render? I’ve been trying to do it using a combination of the below docs -

Background Callbacks | Dash for Python Documentation | Plotly

Deploy a Celery Worker | Render Docs

I’ve added the below lines to my dash app:

if ‘REDIS_URL’ in os.environ:
# Use Redis & Celery if REDIS_URL set as an env variable
print(‘redis_url found in os.environ’)
from celery import Celery
celery_app = Celery(name, broker=os.environ[‘REDIS_URL’], backend=os.environ[‘REDIS_URL’])
background_callback_manager = CeleryManager(celery_app)
print(‘imported celery’)

I see that my logs on Render do show print statements with “imported celery”, so I believe these lines are working. I believe I have correctly started the Redis server.

I am confused on what I need to put for Repo, Build Command, and Start Command for my Render Background Worker. The docs above seem to have slightly conflicting information - The Dash documentation above seems to be suggesting a procfile with the below lines:
web: gunicorn app:server --workers 4
queue: celery -A app:celery_app worker --loglevel=INFO --concurrency=2

Whereas the Render documentation above seems to be suggesting a “Start Command” of
celery --app tasks worker --loglevel info --concurrency 4

I have tried various combinations but I can’t seem to correctly deploy the celery background worker - I get errors when I try to deploy the worker, and my callback just spins indefinitely.

Any help from anyone who has been able to make this work would be much appreciated.

Many thanks for reading!

[Pippo]

I think I have got it working, below I have put my steps in case anyone else struggles with this in future. I can’t be sure that all of these settings (eg my specific Python version, and all of the requirements lines I listed) are absolutely necessary but they are what I have and it seems to be working:

1. Preparing your Dash app and Github repository: Add the background worker lines to your Dash app, as shown in “Example 1” here. Background Callbacks | Dash for Python Documentation | Plotly
   I also added some additional lines to my requirements.txt to make sure the dependencies get installed at Render in the steps below. I added:
   dash[celery]==2.11.1
   celery==5.3.1
   diskcache==5.6.1
   redis==4.6.0
   gunicorn==20.1.0
   Upload this app to a Github repository (it can be public or private).
2. Set up a new Redis service on Render, I believe I left all options at default other than “maxmemory policy” which I changed to “noeviction” as it was recommended for queue services
3. Set up a new Background Worker service on Render, with the below settings. I believe this is the process that executes the code that is within the Background Callback:
   Repository: the URL of your dash app on github from step 1
   Build Command : pip install --upgrade pip setuptools wheel && pip install -r requirements.txt
   Start Command: celery -A app:celery_app worker --loglevel=INFO --concurrency=2
   Environment variable: PYTHON_VERSION = 3.11.4
   Environment variable: REDIS_URL = Use the “internal redis url” found within the info tab of the Redis service you set up in step 2
4. Set up your a new Web Service on render, with the below settings. I believe this is the process that executes the code that is not in the Background Callback:
   Repository: the URL of your dash app on github from step 1
   Build Command : pip install --upgrade pip setuptools wheel && pip install -r requirements.txt
   Start Command: gunicorn app:server
   Environment variable: PYTHON_VERSION = 3.11.4
   Environment variable: REDIS_URL = Use the “internal redis url” found within the info tab of the Redis service you set up in step 2

One of the things that was causing me errors initially was my use of Flask Limiter as a rate limiter, it seems to stop my app from working when I change my standard dash callback to a Background Callback, so I had to disable my Flask Limiter lines. I will make a new topic about this as it would be good to be able to set some kind of rate limit so that the app/queue can’t be overloaded. Edit - here Flask Rate Limiter does not seem compatible with Background Callbacks - any other options?

Many thanks again to Nicholas Kluge for sharing how he got the web service working in the first place