đź“Ł Dash v1.11.0 Release - Introducing Pattern-Matching Callbacks

Update: version 1.16 has been released since this was posted.

Dash v1.11.0 is a minor release introducing the new Pattern-Matching Callbacks feature and providing multiple bug fixes improving the performance, robustness and behavior of Dash callbacks.

Official Changelog
Dash v1.11.0

Highlights

  • New Pattern-Matching Callbacks feature that allows you to write callbacks that respond to or update an arbitrary or dynamic number of components (PR #1103)
  • Multiple bug fixes for callbacks, including bug, bug, bug, bug, bug, and a bug where async components that aren’t rendered by the page (for example in a background Tab) would block the app from executing callbacks
  • dcc.Loading component bug fix where components, mainly graphs with uirevision, would behave incorrectly after multiple updates. (PR#740 )
  • Update dcc.Loading so that components are hidden instead of removed during loading state. Besides fixing the bug with how dcc.Graph & uirevision would behave, it also ensures that the height of the component does not shrink or collapse when it goes into a loading state.

Previous Releases
Dash v1.10.0
Dash v1.9.0
Dash v1.8.0

Documentation Updates
New Pattern-Matching Callbacks Documentation now available!

10 Likes

We are very excited about Pattern-Matching callbacks! This is one of the biggest features & deepest overhauls to Dash’s core architecture that we’ve made since we originally released the framework in 2017.

In August 2017, one month after we released Dash, we opened up a forum topic about one of the framework’s core limitations: Dynamic Controls and Dynamic Output Components. From that thread, we said:

This means that you must generate callbacks for every unique set of input components that could be present on the page.

In Dash, the callback function’s decorator’s aren’t dynamic. Each set of input components can only update a single output component. So, for each unique set of input components that are generated, you must generate a unique output component as well.

This was one of the most viewed & linked-to topics in the forum… a roadblock that many of us ran into when creating more complex Dash applications

The new Pattern-Matching Callbacks feature removes this limitation. It allows you to create callbacks that match a group (or, a “pattern”) of component IDs, rather than just a single callback ID.

The documentation goes into much more detail, but this is just a quick example to give you an idea of what’s possible in less than 100 lines of code:

import dash
from dash.dependencies import Input, Output, State, ALL, MATCH
import dash_html_components as html
import dash_core_components as dcc
import plotly.express as px

df = px.data.gapminder()

app = dash.Dash(__name__)

app.layout = html.Div([
    html.Div(children=[
        dcc.Dropdown(
            options=[{
                'label': i,
                'value': i
            } for i in df.country.unique()],
            value='Canada',
            id='country',
            style={'display': 'inline-block', 'width': 200}
        ),
        html.Button(
            'Add Chart', id='add-chart', n_clicks=0,
            style={'display': 'inline-block'}
        ),
    ]),

    html.Div(id='container', children=[])
])


def create_figure(column_x, column_y, country):
    chart_type = px.line if column_x == 'year' else px.scatter
    return chart_type(
        df.query("country == '{}'".format(country)),
        x=column_x,
        y=column_y,
    )\
    .update_layout(
        title='{} {} vs {}'.format(country, column_x, column_y),
        margin_l=10, margin_r=0, margin_b=30)\
    .update_xaxes(title_text='').update_yaxes(title_text='')

@app.callback(
    Output('container', 'children'),
    [Input('add-chart', 'n_clicks')],
    [State('container', 'children'),
     State('country', 'value')])
def display_dropdowns(n_clicks, children, country):
    default_column_x = 'year'
    default_column_y = 'gdpPercap'

    new_element = html.Div(
        style={'width': '23%', 'display': 'inline-block', 'outline': 'thin lightgrey solid', 'padding': 10},
        children=[
            dcc.Graph(
                id={
                    'type': 'dynamic-output',
                    'index': n_clicks
                },
                style={'height': 300},
                figure=create_figure(default_column_x, default_column_y, country)
            ),
            dcc.Dropdown(
                id={
                    'type': 'dynamic-dropdown-x',
                    'index': n_clicks
                },
                options=[{'label': i, 'value': i} for i in df.columns],
                value=default_column_x
            ),
            dcc.Dropdown(
                id={
                    'type': 'dynamic-dropdown-y',
                    'index': n_clicks
                },
                options=[{'label': i, 'value': i} for i in df.columns],
                value=default_column_y
            ),
        ]
    )
    children.append(new_element)
    return children


@app.callback(
    Output({'type': 'dynamic-output', 'index': MATCH}, 'figure'),
    [Input({'type': 'dynamic-dropdown-x', 'index': MATCH}, 'value'),
     Input({'type': 'dynamic-dropdown-y', 'index': MATCH}, 'value'),
     Input('country', 'value')],
)
def display_output(column_x, column_y, country):
    return create_figure(column_x, column_y, country)


if __name__ == '__main__':
    app.run_server(debug=True)

We were able to do all of this without changing the stateless architecture that makes Dash so special.Multiple people can view this app at the same time and have independent sessions. Scaling this app to 10s, 100s, or thousands of viewers is as simple as adding more workers or nodes.

We’ll be talking a lot more about this feature in the coming weeks, so stay tuned for more announcements. In the meantime, try it out, let us know how it goes, and show us what you’ve made!

10 Likes

Amazing news! This is by far the Dash release that i have been looking forward to the most :smiley:

7 Likes

This is amazing news. Thank you.

In addition, I have one question about app.config.suppress_callback_exceptions.

So I have a multi-page application, I have a callback function1 to generate a barchart with id b1. Then I have a callback function2 that use id b1 as input to get its clickData value. With version 1.11.0, I got the error

'A nonexistent object was used in an Input of a Dash callback. The id of this object is b1 ’

I understood that i was calling the b1 in my callback before the component is defined on initial layout. So I used app.config.suppress_callback_exceptions = True. This is working for the previous version 1.10.0 but not the most recent version. Did we have any changes on this functionality? How should i suppress the exception with 1.11.0?

I have modified the code for pattern-matching callbacks to reproduce my error

import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output, State, MATCH, ALL

app = dash.Dash(__name__, suppress_callback_exceptions=True)

app.layout = html.Div([
    html.Div(id='content'),
    dcc.Location(id='url', refresh=False)
])


def render_layout():
    return html.Div([
        html.Button("Add Filter", id="add-filter", n_clicks=0),
        html.Div(id='dropdown-container', children=[]),
        html.Div(id='dropdown-container-output')
    ])


@app.callback(dash.dependencies.Output('content', 'children'),

              [dash.dependencies.Input('url', 'pathname')],
              [dash.dependencies.State('url', 'href')])
def display_content(url, href):
    if url:
        return render_layout()


@app.callback(
    Output('dropdown-container', 'children'),
    [Input('add-filter', 'n_clicks')],
    [State('dropdown-container', 'children')])
def display_dropdowns(n_clicks, children):
    new_dropdown = dcc.Dropdown(
        id='filter-dropdown',
        options=[{'label': i, 'value': i} for i in ['NYC', 'MTL', 'LA', 'TOKYO']]
    )
    children.append(new_dropdown)
    return children


@app.callback(
    Output('dropdown-container-output', 'children'),
    [Input('filter-dropdown', 'value')]
)
def display_output(values):
    return html.Div([
        html.Div('Dropdown {} = {}'.format(i + 1, value))
        for (i, value) in enumerate(values)
    ])


if __name__ == '__main__':
    app.run_server(debug=True)

Here is the output
image

Thanks for looking into this.

1 Like

Hi @meg - sorry about this, it does look like the behavior around this case changed, though we’re trying to figure out if it should actually be considered a bug. If I take the code you posted and run it in dash 1.10 it looks like it’s OK, no error shows up in the on-page devtools. But there actually is an error in the JS console, very similar to what you’re seeing in v1.11:

I’m curious, if you try your full app with dash 1.10 or before and open the JS console, do you also see a similar error? If you do, then I tend to think we’ve found a case of a previously-swallowed error that’s now being caught, but the behavior after the error is not something we can guarantee won’t change. If you don’t see a console error, then there’s something else going on and we should try to reproduce it more precisely.

Now, as to actually avoiding this error: the suggestion I’d give before v1.11 is to see if component b1 could be inserted in the page at the same time as the component that depends on it, just start it out hidden (style={'display': 'none'}) and have the callback that’s currently generating b1 instead just set its figure prop and un-hide it (style: {})

As of v1.11 you of course have the option of using a dict ID instead of b1, and have the callback that uses it do so using an ALL wildcard. For example the ID could be {'graph': 'b1', 'index': 0} and reference it with Input({'graph': 'b1', 'index': ALL}, <prop>). Then when the callback is invoked initially it will get [] as the input, and [value] when b1 exists.

I should note that this general technique can be used to easily share data between pages of a multi-page app: make one dcc.Store at the top level, and collect data into it from different pages using components with ID pattern that you combine with an ALL wildcard.

2 Likes

I am also seeing this issue in one of my full apps. The weird thing is that i have numerous callbacks linked to components that do do not exist, but only a few of them are throwing the error. For now, i have used wildcards to silence the error for these components,

It seems that the fix i came up with is essentially equal to the suggestion by @alexcjohnson :smile: . I am slightly split about it, though i like it much better than the toggle-visible-fix. From a logical point of view, it seems reasonable that a wildcard callback should be used whenever the number of components is not constant, which is in fact the case; there can be zero or one component. However, the syntax is more complex for wildcard ids, and it worked just fine before without. As this zero-or-one case is rather common, a simple syntax would be preferable.

To my understanding, in Dash 1.10.0 (and lower), a callback was simply not fired, if none of the input elements were present. Is this wrong? And if it is true, is there any reason not just to do this again?

After updating to 1.11.0, i have also started seeing the following warning in the js console,

react-dom@16.v1_3_0m1586552074.13.0.js:82 Warning: Cannot update a component from inside the function body of a different component.

The app seems to work fine though.

1 Like

@Emil same question I asked @meg - if you run this app with v1.10, do you see any errors in the JS console? My understanding is these callbacks would always have tried to fire, and it was always supposed to be an error to have partially-defined callbacks. But as the example Meg posted shows, we were previously swallowing some of these errors so you’d only see them from the JS console.

If that’s NOT the case I’d love to see a simplified example: an app that worked with no errors anywhere in 1.10 but throws an error in 1.11.

Cannot update a component from inside the function body of a different component.

I saw that at some point as well but haven’t tracked it down. Do you know if this error started showing up in 1.11 or if it was already there in v1.10? I was going to guess that it was introduced by the React upgrade (v16.8->v16.13) we included in v1.10, where React is getting more opinionated to prepare folks for changes in their upcoming major v17 release.

Hi all,

I have a similar problem as @meg and it’s clearly a 1.11.0 issue (doesn’t appear in 1.10.0).

It appears within a callback that is fired on instantiating the app. It dynamically creates some components, registers some callbacks and then returns the components in the layout (as a html.DIV’s children).

Another issue, I’ve been having when trying to run the app with v1.11.0 is I can’t anymore try / catch a DuplicateCallbackOutput error (doesn’t exist anymore and is now dealt with by the browser as far as i’ve understood).

Anyway, the Pattern Matching Callbacks looks awesome, hope I’ll be able to solve these issues i’ve mentionned and then use this pattern !

Hi @alexcjohnson. Thanks for the reply.

When I use 1.10 in my full app, I didn’t see ReferenceError.

image

I believe your suggestions on the work around should solve this problem. Regardless of whether the error is new, is this going to be an open issue and plan to resolve in future releases?

Thanks

OK good to know. I would love to try and pin down exactly the conditions under which Dash 1.10 does not throw any error but Dash 1.11 does. If anyone can find a case of this they can share I’d be very grateful!

Anyway this was always supposed to be an error, and it seems like (as of Dash 1.10) small changes in app structure can switch it between error and not. So unless we can find a very clear and robust criterion for allowing it to pass I’d be inclined to continue treating it as an error - just being more consistent about it now.

We have had some discussions internally about possibly marking certain inputs as optional - kind of a special mini version of pattern-matching specifically for the zero-or-one case. But that’s not on the roadmap yet and will take some more discussion and a decent amount of work.

Interesting, can you say more? What were you doing in the try/except with DuplicateCallbackOutput? There would be workarounds with app.callback_map but not as simple as simply trapping the correct error class.

Sorry, wasn’t online for a few days.

So, here’s the idea. I have a callback that is triggered when the user opens a page. Depending on the user role, I render some components and register their callbacks. All of this is done in this callback triggered on opening the page.

def render_allowed_components(_):
	"""
	Returns the correct components that the logged in user can access and register the callbacks for these components

	"""
	# import here to avoid circular imports + better performance (avoid import useless component)
	from app.pages.main_dashboard.children import children
	return filter_register_accessible_comps(current_user, children)

The filter_register_accessible_comps method looks like this :

def filter_register_accessible_comps(current_user, components):
	"""
	From a list of children components, we filter out those that the current_user is not allowed to access
	If user is allowed to access a child component (user.role <= child.allowed_role): register callbacks + keep th child
	Otherwise : DONT register callbacks + replace child by empty component 
	"""
	empty_component = ""
	def filter_register(child):
		
		if child.allowed_role >= current_user.role:
			child.register_cbs()
			return child.layout()
		else:
			return empty_component

	return list(map(lambda child: filter_register(child), components))

One of my issue then is a user logs in, opens this page and reloads it (or just switches to another page and comes back), callbacks would already be registered with the same outputs / inputs. I just need to return the right components without registering an already existing callback. So I have this line in the register_cbs method to catch and ignore this error :

try:
		cb = create_callback(outputs, fn)
		app.callback(output=outputs, inputs=inputs, state=states)(cb)
	except DuplicateCallbackOutput:
		logger.warning(f"Error registering callback DuplicateCallbackOutput {fn} {inputs}")
		pass
	except CallbackException:
		logger.warning(f"Other Callback Exception registering cb for {fn} {inputs}")
		pass
	except Exception as e:
		exc_type, exc_obj, exc_tb = sys.exc_info()
		logger.error(f"Unexpected error: {exc_type} {exc_obj} {exc_tb}")
		pass

So, with dash 1.11.0 - unlike v1.10 - the app.callback(…) line doesn’t raise any error so I can’t catch it. I would get these error messages and it would block my app :

In the callback for output(s):
********
*********
Output * (***.*) is already in use.
Any given output can only have one callback that sets it.
To resolve this situation, try combining these into
one callback function, distinguishing the trigger
by using dash.callback_context if necessary.

Interesting, I see. Honestly I’m a bit surprised that would work even in previous versions. For one thing, how do these new callbacks make their way to the page? Once you’re at the point of executing callbacks, the page has all the callbacks it’s ever going to get without a complete reload - same with importing new components from inside a callback. And then if another user comes along and has lesser permissions, the app has already loaded the greater callbacks and component packages so this new user will get them too - these components just won’t be displayed, as they’re not returned by render_allowed_components.

One of the key features of Dash is that the back end is stateless - it doesn’t hold any information about how it’s being used in the browser. That means each request needs to function independently, deriving its entire behavior from just the inputs received in that request, and nothing else in the back end (in this case callback registration and component loading) is affected by the request. Among other things, that allows your app to serve multiple users simultaneously, or have the same user served by multiple independent copies of the app.

The way we would typically advise building an app like this is to define all possible callbacks up front, but then check credentials within each callback to determine if the user for that particular request is allowed to perform that function. Then if the components a user is not intended to see don’t connect to components they CAN see (via the full set of callbacks), you can leave these components out of the layout entirely. If they DO connect to visible components, up to v1.10 you might need to insert hidden copies. As of v1.11 though, you can still omit them, if you use pattern-matching callbacks with the ALL wildcard to match zero or one item.

2 Likes

Hi @alexcjohnson - the following app works fine with Dash 1.10, but shows a ReferenceError in the JS console for v1.11:

import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output, State


app = dash.Dash(__name__)
app.config.suppress_callback_exceptions = True
app.layout = html.Div([
    dcc.Location(id='url', refresh=False),
    html.Div(id='content')
])


@app.callback(Output('content', 'children'),
              [Input('url', 'search')],
              [State('content', 'children')])
def initialize_from_url(search, content):
    if content:
        raise dash.exceptions.PreventUpdate
    foobar_value = [x for x in ('foo', 'bar') if x in search]
    return serve_content(foobar_value)


def serve_content(foobar_value):
    content = [
        dcc.Checklist(
            id='foobar',
            options=[{'value': 'foo', 'label': "Foo"},
                     {'value': 'bar', 'label': "Bar"}],
            value=foobar_value),
    ]
    return content


@app.callback(Output('url', 'search'), [Input('foobar', 'value')])
def update_url(foobar_value):
    return '?' + '&'.join(foobar_value) if foobar_value else ''


app.run_server(debug=True)

Hey @alexcjohnson, thanks for the reply. I’ve looked more into the logic of the code and it seems that you’re right. All the callbacks were registered anyway on the start of the app and then that filter_register_accessible_comps was actually only useful for returning the allowed components’ layouts (the dynamic callback registration generated a CallbackException each time and was completely useful).

I’ll implement the logic you’ve detailed, sounds good. And then I’ll probably be able to update to v1.11 which would be great.

I’ll put the code here

1 Like

Thanks everyone who’s weighed in and helped us nail down the change of behavior between v1.10 and v1.11 with callbacks having (initially) outputs on the page but no inputs. After a bunch of internal discussions, we’ve decided to try and treat this change as a regression. If your app is impacted by this, I’d encourage you to stay on v1.10 for now, and watch for v1.12 in which we will do our best to include a fix. We’ll try to get this resolved in the next week or so.

The proposed rule is:

If:

  • None of the inputs for a certain callback are on the page
  • The inputs are NOT entirely multi-item patterns (with ALL or ALLSMALLER wildcards)

Then:

  • We will not raise an error
  • We will not try to fire this callback
  • If any of the outputs of this callback are themselves inputs to other callbacks, those other callbacks will fire immediately.

We have some concerns about this, but this behavior is clearly filling an important need for certain use cases so the best thing to do is document these caveats but restore the behavior. Anyway these concerns apply at least as much to the old behavior, we’re just trying to make it all as clear and predictable as possible.

We’re mainly worried about the last point about callbacks using the output of the un-called callback as an input. The first issue is the potential for calling these callback with None or other unexpected inputs. That one should show up during development, so if it leads to an error you’ll be alerted to it and can fix it. More subtle is the fact that as soon as the inputs to the first callback arrive, the latter callback (and any others chained off it) will be ignored, but this could happen anywhere in the chain, leading to unpredictable behavior and especially differences between dev and production (where network lag may be greater, for example). The end state should be the same in all cases, so in principle this only impacts what the user sees during loading, but in certain cases that intermediate state will look inconsistent.

1 Like

Great detective work @alexcjohnson! I was not aware that this was in fact how Dash was behaving prior to 1.11.0. I just through that callbacks were silenced, when none of the inputs were present. The behavior that you sketch seems rather incoherent.

With the Dash framework being as youg that it is, i would definetly opt for making Dash as coherent as possible, even it that means breaking backward compability in a few corner cases. You can’t make a (good) omelette without breaking an egg :slight_smile:

2 Likes