🚀 Gen 5 of the leading AI app deployment platform launches October 6. Click for the livestream.

Dash 2.0 + Celery, Heroku Procfile config to register long_callback? - Error: Received unregistered task of type 'long_callback_[hash]'

Trying to use celery for the first time on Heroku in a Dash 2.0 app to run a long-running function. I think my Procfile is incorrect but I am not sure what it should be.

I am using a multi page app structure. My app.py contains my app variable and is where I register the long callback manager.
Procfile
This generates the error:

web: gunicorn index:server
worker: celery --app=app.long_callback_manager worker --loglevel=info

My app.py:

from dash.long_callback import CeleryLongCallbackManager
from celery import Celery

celery_app = Celery(__name__, broker=os.environ['REDIS_URL'], backend=os.environ['REDIS_URL'])
long_callback_manager = CeleryLongCallbackManager(celery_app)
 
### ... ###

app = Dash(__name__,
           external_stylesheets=[dbc.themes.COSMO],
           long_callback_manager=long_callback_manager,
           suppress_callback_exceptions=True)

When trying to execute a long_callback, which seems to be supplying the right info, I get the error:


2021-08-26T20:26:51.845762+00:00 app[worker.1]: [2021-08-26 20:26:51,845: ERROR/MainProcess] Received unregistered task of type 'long_callback_61ab3ce7fae
e53c1837008c09a44970689f42580'.
2021-08-26T20:26:51.845768+00:00 app[worker.1]: The message has been ignored and discarded.
2021-08-26T20:26:51.845769+00:00 app[worker.1]:
2021-08-26T20:26:51.845769+00:00 app[worker.1]: Did you remember to import the module containing this task?
2021-08-26T20:26:51.845770+00:00 app[worker.1]: Or maybe you're using relative imports?
2021-08-26T20:26:51.845770+00:00 app[worker.1]:
2021-08-26T20:26:51.845770+00:00 app[worker.1]: Please see
2021-08-26T20:26:51.845770+00:00 app[worker.1]: http://docs.celeryq.org/en/latest/internals/protocol.html
2021-08-26T20:26:51.845771+00:00 app[worker.1]: for more information.
2021-08-26T20:26:51.845771+00:00 app[worker.1]:
2021-08-26T20:26:51.845771+00:00 app[worker.1]: The full contents of the message body was:
2021-08-26T20:26:51.845772+00:00 app[worker.1]: b'[["29d8b24e563abcff50fd7886875909aea755e7aa", "29d8b24e563abcff50fd7886875909aea755e7aa-progress", [...MY PAYLOAD HERE...]], {}, {"callb
acks": null, "errbacks": null, "chain": null, "chord": null}]' (319b)
2021-08-26T20:26:51.845772+00:00 app[worker.1]: Traceback (most recent call last):
2021-08-26T20:26:51.845773+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 581, in on
_task_received

EDIT:
I should mention tried worker: celery --app=app.celery_app worker --loglevel=info but that didnt work as well. It also says it received and unregistered task type.

I also tried worker: celery --app=app.long_callback_manager worker --loglevel=info

With that it appears to be failing on startup with the error:
2021-08-26T20:54:55.440500+00:00 app[worker.1]: AttributeError: 'CeleryLongCallbackManager' object has no attribute 'user_options'

EDIT2: Looks like it might be due to no importing the decorated function from my other .py file? Not sure how to go about that.

Went on vacation and came back at this with fresh eyes. Still having the issue though. I think this may be related to trying to use it in a multipart app, similar to the issue I had here.

That Heroku is spitting out 2021-09-13T15:48:03.790014+00:00 app[worker.1]: [2021-09-13 15:48:03,789: ERROR/MainProcess] Received unregistered task of type 'long_callback_61ab3ce7f aee53c1837008c09a44970689f42580'. in the logs indicates to me that the problem is the lack of presence of the callback. I’m not sure how to get it to import it though.

So this isn’t “Solved” but I think I am giving up on long_callbacks for now and instead just going to poll a lambda function. I did however get a fix for the unregistered error by adding an include param and setting autodiscover

For ref my @long_callback decorated function is in apps/video.py:

    celery_app = Celery(__name__,
                        broker=os.environ['REDIS_URL'],
                        backend=os.environ['REDIS_URL'],
                        include=['apps.video']
                        )
    celery_app.autodiscover_tasks(['apps'])

There seems to be a bunch of weird background things happening with the task failing on heroku, memory limits getting exceeded, the whole thing is a PITA and it’ll likely be way easier to just use an AWS Lambda function.

Thanks for reporting @caseguide! We’re taking a look at this.