Celery not registering background callbacks in task list

Dash Python
1

I want to use background callbacks and individual celery tasks with the same Celery queue.

My current structure is the following to avoid circular imports:

/
-app.py (creates Dash app, imports celery_manager)
-app_global.py (created celery_app and celery_manager. celery tasks defined here)
-app_callbacks.py (has a callback that adds tasks to celery queue with celery_app object)
-pages/page.py (has a background callback, imports app but not celery_app, background=True)

By having celery tasks defined in app_global.py where the celery_app instance is defined, they are registered in the celery tasks dictionary, but Dash’s background_callback’s aren’t registered and I get the error:

“Received unregistered task of type ‘long_callback_5abe53881a52c2a77290c5e53062ad47fbbd1e18’.”

I’m looking for any help to solve this. My current ideas are (1) background callbacks and celery tasks shouldn’t be in the same queue, and (2) the app = Dash() object needs to be instantiated in the same file as celery_app = Celery().

1 Like
2

Is this work being done on a public repo?

3

Yeah, here:

Will push more recent changes, sketch of files in original post was a simplification but is pretty close.

4

Hey mate,

Just glanced at your celery init, technically “__name__” should be __name__

Did you try to bring celery and the background manager init into the app and run it from there?

5

Yeah I did, that’s a part of my confusion. Celery isn’t recognizing any background callbacks across my application. It also lead to circular import problems.

6

Problem persists even if I specify the CeleryManager instance in my callback’s decorator.

7

Okay problem solved, for future docs for others here was the problem:

When I started the celery workers, I was referencing the ‘app_global’ file like so:

celery -A app_global:celery_app worker --loglevel=DEBUG --concurrency=2

What I should have done was import the celery_app from app_global into app.py and run the worker like this:

celery -A app:celery_app worker --loglevel=DEBUG --concurrency=2

This solved my problem immediately.

Thank you @jinnyzor and @cdolfi for the help along the way, I improved my code because of your input.

2 Likes
8

Other food for thought, please include a requirements.txt on your repo. :wink:

9

Thanks for the suggestion and help as always! Will do!

1 Like