Dash 2.0 + Celery, Heroku Procfile config to register long_callback? - Error: Received unregistered task of type 'long_callback_[hash]'

Dash Python
1

Trying to use celery for the first time on Heroku in a Dash 2.0 app to run a long-running function. I think my Procfile is incorrect but I am not sure what it should be.

I am using a multi page app structure. My app.py contains my app variable and is where I register the long callback manager.
Procfile
This generates the error:

web: gunicorn index:server
worker: celery --app=app.long_callback_manager worker --loglevel=info

My app.py:

from dash.long_callback import CeleryLongCallbackManager
from celery import Celery

celery_app = Celery(__name__, broker=os.environ['REDIS_URL'], backend=os.environ['REDIS_URL'])
long_callback_manager = CeleryLongCallbackManager(celery_app)
 
### ... ###

app = Dash(__name__,
           external_stylesheets=[dbc.themes.COSMO],
           long_callback_manager=long_callback_manager,
           suppress_callback_exceptions=True)

When trying to execute a long_callback, which seems to be supplying the right info, I get the error:


2021-08-26T20:26:51.845762+00:00 app[worker.1]: [2021-08-26 20:26:51,845: ERROR/MainProcess] Received unregistered task of type 'long_callback_61ab3ce7fae
e53c1837008c09a44970689f42580'.
2021-08-26T20:26:51.845768+00:00 app[worker.1]: The message has been ignored and discarded.
2021-08-26T20:26:51.845769+00:00 app[worker.1]:
2021-08-26T20:26:51.845769+00:00 app[worker.1]: Did you remember to import the module containing this task?
2021-08-26T20:26:51.845770+00:00 app[worker.1]: Or maybe you're using relative imports?
2021-08-26T20:26:51.845770+00:00 app[worker.1]:
2021-08-26T20:26:51.845770+00:00 app[worker.1]: Please see
2021-08-26T20:26:51.845770+00:00 app[worker.1]: http://docs.celeryq.org/en/latest/internals/protocol.html
2021-08-26T20:26:51.845771+00:00 app[worker.1]: for more information.
2021-08-26T20:26:51.845771+00:00 app[worker.1]:
2021-08-26T20:26:51.845771+00:00 app[worker.1]: The full contents of the message body was:
2021-08-26T20:26:51.845772+00:00 app[worker.1]: b'[["29d8b24e563abcff50fd7886875909aea755e7aa", "29d8b24e563abcff50fd7886875909aea755e7aa-progress", [...MY PAYLOAD HERE...]], {}, {"callb
acks": null, "errbacks": null, "chain": null, "chord": null}]' (319b)
2021-08-26T20:26:51.845772+00:00 app[worker.1]: Traceback (most recent call last):
2021-08-26T20:26:51.845773+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 581, in on
_task_received

EDIT:
I should mention tried worker: celery --app=app.celery_app worker --loglevel=info but that didnt work as well. It also says it received and unregistered task type.

I also tried worker: celery --app=app.long_callback_manager worker --loglevel=info

With that it appears to be failing on startup with the error:
2021-08-26T20:54:55.440500+00:00 app[worker.1]: AttributeError: 'CeleryLongCallbackManager' object has no attribute 'user_options'

EDIT2: Looks like it might be due to no importing the decorated function from my other .py file? Not sure how to go about that.

2

Went on vacation and came back at this with fresh eyes. Still having the issue though. I think this may be related to trying to use it in a multipart app, similar to the issue I had here.

That Heroku is spitting out 2021-09-13T15:48:03.790014+00:00 app[worker.1]: [2021-09-13 15:48:03,789: ERROR/MainProcess] Received unregistered task of type 'long_callback_61ab3ce7f aee53c1837008c09a44970689f42580'. in the logs indicates to me that the problem is the lack of presence of the callback. I’m not sure how to get it to import it though.

3

So this isn’t “Solved” but I think I am giving up on long_callbacks for now and instead just going to poll a lambda function. I did however get a fix for the unregistered error by adding an include param and setting autodiscover

For ref my @long_callback decorated function is in apps/video.py:

    celery_app = Celery(__name__,
                        broker=os.environ['REDIS_URL'],
                        backend=os.environ['REDIS_URL'],
                        include=['apps.video']
                        )
    celery_app.autodiscover_tasks(['apps'])

There seems to be a bunch of weird background things happening with the task failing on heroku, memory limits getting exceeded, the whole thing is a PITA and it’ll likely be way easier to just use an AWS Lambda function.

4

Thanks for reporting @caseguide! We’re taking a look at this.

1 Like
6

@chriddyp Was there any resolution to this, I have exactly the same problem. I also agree with @caseguide that something is happening with the memory limits being exceeded.

1 Like
7

Wondering if there is any further update to this question? Celery isn’t recognizing my background callbacks either.

8

I’m having an identical problem. Has anyone managed to find a solution?

9

I’m also facing this problem. Was there a solution to this @chriddyp ?

10

I’m a little late but hopefully this post will help someone in the future:

My guess is that the celery manager is defined in app.py but the long callback is defined somewhere else that isn’t imported into app.py and so, since the entry point for the celery command is in app.py, celery isn’t aware of that task after running celery -A app:celery_app worker --loglevel=DEBUG --concurrency=2. The solution would be change the celery command’s entry point to a file that is aware of the long callback.

E.g if you have a multipage app where the dash app /celery_app objects are defined in app.py but then the long callback is used in the pages/home.py file of a multipage app, you need to make sure that you run the celery command from a file that imports pages.home . So if index.py does this and the app /celery_app objects are also imported into it from app.py, then you can run celery using celery -A index:celery_app worker --loglevel=DEBUG --concurrency=2 instead of celery -A app:celery_app worker --loglevel=DEBUG --concurrency=2

This way, the long callback will be registered and you will see something like this in your celery logs when it start up showing that the long callback is indeed registered:

[tasks]
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . long_callback_70c4bb3bf988fc737bf70891d98c68aa4beb089a
2 Likes