[dash 2.9.2] KeyError: "Callback function not found for output does not make any sence in Docker

Dash Python
1

So, long story short, I have a Dash app inside of a bigger Flash app. On one point, I define a function that would be called by init routines and define callbacks.
Smth like this (in an abridged form):

def make_callbacks(dashapp):
    @dashapp.callback(
        Output('graph_impl', 'elements'),
        Output('download_button', 'disabled', allow_duplicate=True),
        Output('graph_info', 'children', allow_duplicate=True),
        Input('graph_select', 'value'),
        prevent_initial_call=True,
    )
    def select_graph(sel_opt):
        logging.info("----------- SELECT")

Here, graph_impl is a cytograph, download_button, well, button, graph_select is dcc.Dropdown, and graph_info is just a Div. But all that does not seem to make any difference.

When I run it via plain Interpreter on bare server everything works fine. When I run it with gunicorn on bare server everything works fine.

But when I try to run it inside Docker I am getting the message from the topic name when try to switch options in that Dropdown. And that makes zero sense for me – it is clearly written correctly, moreover the logic works fine outside Docker.

I was just wondering:

  1. Is there any peculiar behavior of Dash specifically when it comes to Docker (which I fairly really doubt);
  2. What that error even suppose to mean? I’ve seen tons of weird situation definitely not related to the issue stated in the description. Any I sure have no idea why that does not work within Docker specifically.

As for a bit of a context – I have a redis in a separate service and a plain sqlite db mapped through volume – but they work correctly, both instantiated and then accessed from inside the container in question.

I would greatly appreciate any assistance in the matter, and any idea for the possible culprit.

As for the full error description – please find it below:

ERROR:app:Exception on /graph/_dash-update-component [POST]
Traceback (most recent call last):
File “/usr/local/lib/python3.10/site-packages/dash/dash.py”, line 1230, in dispatch
cb = self.callback_map[output]
KeyError: ‘…graph_impl.elements…download_button.disabled@bedc8f5f5f124508b93a7d5819d8c8d8…graph_info.children@69ff1ba782d549d09d66a07e0e62d8fe…’

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/usr/local/lib/python3.10/site-packages/flask/app.py”, line 2528, in wsgi_app
response = self.full_dispatch_request()
File “/usr/local/lib/python3.10/site-packages/flask/app.py”, line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File “/usr/local/lib/python3.10/site-packages/flask/app.py”, line 1823, in full_dispatch_request
rv = self.dispatch_request()
File “/usr/local/lib/python3.10/site-packages/flask/app.py”, line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File “/usr/local/lib/python3.10/site-packages/flask_login/utils.py”, line 290, in decorated_view
return current_app.ensure_sync(func)(*args, **kwargs)
File “/usr/local/lib/python3.10/site-packages/dash/dash.py”, line 1279, in dispatch
raise KeyError(msg) from missing_callback_function
KeyError: “Callback function not found for output ‘…graph_impl.elements…download_button.disabled@bedc8f5f5f124508b93a7d5819d8c8d8…graph_info.children@69ff1ba782d549d09d66a07e0e62d8fe…’, perhaps you forgot to prepend the ‘@’?”

2

Having the same problem, no idea on solution but what I think is going on to answer your question #2 is that something about the deployment is creating anonymized callbacks in an attempt to distinguish between multiple instances, but the left hand doesn’t know what the right is doing for some reason so the anonymization isn’t getting passed along.

This is happening to me on an app that worked just fine at Dash 2.3.1, somewhere from 2.3.1 to 2.9.1 the behavior must have changed – though I’m not seeing in the changelog what that might be.

3

Hey, thank for stopping by and welcome to the community!

It turns out that you gave almost a Schrodinger’s answer! Weirdly enough, everything works as it supposed to when the app is run containerized via gunicon with a single worker or with a plain interpreter. But the odd part of that revelation is that there seem to be no problem whatsoever having multiple gunicorns workers running the app outside a container – for some reason multiple workers seem to fail only inside docker.

And what’s about your setup? Are you having these issues with docker too?

4

Yep, also Docker and Gunicorn – but unfortunately I inherited that setup, so there are decisions made in the config that I don’t yet fully understand.

I’ve been playing around with the PYTHONHASHSEED param in Docker because I thought that callback name hash salting could have been the problem, but so far that hasn’t fixed it.

5

Well, I inherited my thing from myself only, haha)

Were you able to reproduce the problem in a testing setup? Just a plain Dash app with a couple of callbacks inside docker? Weirdly enough I’m failing to reproduce this gunicorn problem with trivial layouts – treacherously it performs well with multiple workers not crushing callbacks.

6

No, I’ve not been able to reproduce anywhere except the targeted end deployment (which is the place I’d most like it to work…) yet.

7

Ok, I did solve it. The problem was PYTHONHASHSEED, should be set to 0 (a constant might work too, but random [the default] causes the problem). The reason I didn’t find this at first is because I was setting that value too early in my Dockerfile. Moving PYTHONHASHSEED=0 after a couple of pip operations that must have been resetting it gets me on to my next error.

8

Hey, great that you managed to solve your issue! Doesn’t seem to fix my problem though, but other people might find it useful, so thanks for sharing!

9

Hey. I was facing a similar problem when deploying a Dash application to Elastic Beanstalk with NGINX and gunicorn. The solution for me was upgrading to Dash 2.13.0.

I’m placing my investigation and solution here to help anyone in a similar situation.

My original problem was with Dash 2.9.1 and was originating when using callbacks with duplicate outputs.

The application was working fine when running locally, but when deployed using multiple gunicorn workers it was raising exceptions.

What was weird is that the error was appearing randomly. Sometimes the callbacks were triggered successfully and sometimes they were raising KeyError: “Callback function not found for output".

In my investigation I managed to find out that:

  • The error was happening only for callbacks that had duplicate outputs, with allow_duplicate=True
  • The error was raised only from specific gunicorn workrers, while some were successfully executing the callback
  • When exploring the app.callback_map of the different workers I found out that the outputs with allow_duplicate=True had different component ids, leading to a different callback_id for the same callback in each of the gunicorn workers
  • The reason for that was that dash._utils.create_callback_id from version 2.9.1 was adding a random uuid for each Output with allow_duplicate=True, so when called for the same callback on different workers it was creating a different callback_id
  • The different callback_id meant that only one worker was able to find the proper method in the app.callback_map so unless the request went to it it would raise the exception

This has been fixed in the following commit: Deterministic duplicate outputs. · plotly/dash@1cae404 · GitHub

Therefore updating the Dash version solved the problem for me.

2 Likes
10

Hi, I’m facing the same type of Issue:

I get the same error despite my application working locally even when launched with several gunicron workers but once deployed it logs these errors.
Yet my application operates smoothly.
I’m also using several callbacks with the same output. I’m on Dash 2.13.0 so that doesn’t appear to solve my problem since I also get this error. It might be because I’m using Dash-extesions 1.0.3 with the MultiplexerTransform() enabled (predecessor to allow_duplicate=True). While I have several functions that update the same outputs, there seems to be only 2 functions that trigger these errors when deployed (very weird)

When looking into what dash-extension does with MultiplexerTransform(), it manually applies this duplicate output = True to each callback so I’m confused as to why my error is raised:

11

Try logging out the app.callback_map.keys() and include the process IDs to see what callbacks does each gunicorn worker have.
Note that I also was using dash_extensions so for me the setup to log the keys was:

app = DashProxy()
server = app.server

app.register_callbacks()
logger.info(app.callback_map.keys())

If you add app.register_callbacks() you can see the created callback_map but the app will raise ‘Duplicate callbacks found’. However, in the logs you can still see what each gunicorn worker gets as a callback_map and maybe figure out where the initial KeyError error occurs.

1 Like
12

When doing this in debug mode, I get Output 1 (output.data) is already in use.
To resolve this, set allow_duplicate=True on
duplicate outputs, or combine the outputs into
one callback function, distinguishing the trigger
by using dash.callback_context if necessary.

which I find to be quite out since I use the following options

transforms=[ServersideOutputTransform([cache]),MultiplexerTransform()]

I will try manually specifying allow_duplicate=True

EDIT: Some of these errors are only because of initial callbacks. I’m truly confused as to why only 1 callback generates an error when running despite getting 37 errors in debug mode