How to update single output from variable number of ServersideOutput stores using pattern matching

Hi,

Thank you in advance for any support.

I am trying to create a callback which updates a single Output when any one of a number of ServersideOutput stores are updated. I have successfully used the ALL pattern matching callback to do this with a regular ‘Store’ component but there’s a problem when I try to use ServersideOutput.

What I see is, instead of returning the data, it returns a hash (like what the resulting filename would be when stored) but the file is never written to the local ‘file_system_store/’ folder.

Is this a bug or is there a way to alter my code to achieve what I want?

Full minimal app code pasted below:

from dash import html, dcc, callback, Input, Output, ctx
from dash_extensions.enrich import (
    callback,
    Output,
    Input,
    ServersideOutput,
    html,
    dcc,
    DashProxy,
    ServersideOutputTransform,
)
from dash.dependencies import MATCH, ALL


app = DashProxy(__name__, compress=True, transforms=[ServersideOutputTransform()])

app.layout = html.Div(
    [
        html.Button("A", {"type": "button", "index": "A"}),
        html.Button("B", {"type": "button", "index": "B"}),
        
        dcc.Store(id={"type": "store", "index": "A"}, data=[]),
        dcc.Store(id={"type": "store", "index": "B"}, data=[]),
        
        dcc.Store(id={"type": "serverside-store", "index": "A"}, data=[]),
        dcc.Store(id={"type": "serverside-store", "index": "B"}, data=[]),
        
        html.H1("Serverside storage data using MATCH:"),
        html.Div(id={"type": "serverside-store-output", "index": "A"}, children=[]),
        html.Div(id={"type": "serverside-store-output", "index": "B"}, children=[]),
        
        html.H1("Regular storage data using ALL:"),
        html.Div(id="many-to-one-output", children=[]),
        
        html.H1("Serverside storage data using ALL:"),
        html.Div(id="many-to-one-serside-output", children=[]),
    ]
)


# Regular Store
@callback(
    Output({"type": "store", "index": MATCH}, "data"),
    Input({"type": "button", "index": MATCH}, "n_clicks"),
    prevent_initail_call=True
)
def store(clicks):
    button_id = ctx.triggered_id if not None else "No clicks yet"
    return f"{button_id} : {clicks}"


# Serverside Store
@callback(
    ServersideOutput({"type": "serverside-store", "index": MATCH}, "data"),
    Input({"type": "button", "index": MATCH}, "n_clicks"),
    prevent_initail_call=True,
)
def serverside_store(clicks):
    button_id = ctx.triggered_id if not None else "No clicks yet"
    return f"{button_id} : {clicks}"


@callback(
    Output({"type": "serverside-store-output", "index": MATCH}, "children"),
    Input({"type": "serverside-store", "index": MATCH}, "data"),
    prevent_initail_call=True,
)
def serverside_output(data):
    return data


# many-to-one
@callback(
    Output("many-to-one-output", "children"),
    Input({"type": "store", "index": ALL}, "data"),
    prevent_initail_call=True,
)
def many_to_one_output(data):
    return data


# many-to-one from serverside storage
@callback(
    Output("many-to-one-serside-output", "children"),
    Input({"type": "serverside-store", "index": ALL}, "data"),
    prevent_initail_call=True,
)
def many_to_one_output(data):
    return data


if __name__ == "__main__":
    app.run_server(debug=True)

Hi, dash_extensions is a creation of @Emil
He’ll give you the correct answer :grinning:

Sorry for the long wait time - I am on parental leave, and my new son doesn’t leave me much time for the PC :slight_smile:

To your question; the behaviour you are seeing is a limitation of the current implementation. The implementation was originally made before pattern matching callbacks, which is why edge cases like yours is not handled. I’ll take a look at it when I find some time :slight_smile:

4 Likes

I am working on a new release, which depends on the new (awesome!) Dash 2.9 release. I have also added some improved handling of wildcard callbacks. Could you try if it resolves your issue(s)? To install it, simply run

pip install dash-extensions==1.0.0rc1

NB: This release does not work with Dash 2.8 (or older)

2 Likes

That works! Thank you for your time on this. I’ve only tested this with the example code I posted above but it works there.

Do you know when this might make it into a production release?

That’s great to hear! I’ll do a bit more testing and refactoring. I expect to push a production release with the next weeks.

!!! UPDATED RESOLVED!!! - resolved with dash 1.0.0 and Serverside function (breaking changes)

|| ORIGINAL POST ||
Congrats on the newborn @Emil
The case above works however I’ve found 2 more pattern matching + serverside output edge case bugs:

  1. input matches ALL, output is single output - incorrectly tries to json serialize the serverside output data
  2. non pattern matching use of input from a previously pattern matched serverside output gets hashed into a string.

let me know if you’d like me to raise an issue on github instead

Reproduce code below:

import pandas as pd
from dash_extensions.enrich import (
    callback,
    Output,
    Input,
    ServersideOutput,
    html,
    dcc,
    DashProxy,
    ServersideOutputTransform,
    MATCH, ALL
)

app = DashProxy(__name__, compress=True, transforms=[ServersideOutputTransform()], prevent_initial_callbacks=True)

app.layout = html.Div(
    [
        html.Button("A", {"type": "button", "index": "A"}),
        html.Button("B", {"type": "button", "index": "B"}),

        dcc.Store(id={"type": "serverside-store", "index": "A"}),
        dcc.Store(id={"type": "serverside-store", "index": "B"}),
        dcc.Store(id="serverside-store-all"),

        html.H1("Serverside storage data using MATCH:"),
        html.Div(id={"type": "serverside-store-output", "index": "A"}),
        html.Div(id={"type": "serverside-store-output", "index": "B"}),

        html.H1("Serverside storage data using ALL:"),
        html.Div(id="many-to-one-serverside-output"),
        html.Div(id="output"),
    ]
)


# Serverside Store
@callback(
    ServersideOutput({"type": "serverside-store", "index": MATCH}, "data"),
    Input({"type": "button", "index": MATCH}, "n_clicks"),
)
def serverside_store(clicks):
    res = {'a': [1, 2, 3]}
    return {'aaa': pd.DataFrame(res)}

@callback(
    ServersideOutput("serverside-store-all", "data"),
    Input({"type": "serverside-store", "index": ALL}, "data"),
)
def this_unexpectedly_throws_json_serialization_error(data):
    """
    Combining ALL into a single output causes json serialization error
    """
    return data #TypeError: Object of type DataFrame is not JSON serializable

@callback(
    Output("output", "children"),
    Input({"type": "serverside-store", "index": "A"}, "data"),
)
def this_munges_the_data_into_a_string(data):
    """
    an explicitly selected index causes the input to turn into a hashed string
    """
    return 'this gets munged into: ' + data

@callback(
    Output("many-to-one-serside-output", "children"),
    Input({"type": "serverside-store", "index": ALL}, "data"),
)
def this_works(data):
    return str(data)


if __name__ == "__main__":
    app.run_server()
1 Like

@someaveragepunter Thanks! Let me know if you encounter issues with the new version :slight_smile:

Thanks @Emil ,
I however believe I’ve found another bug:
log transforms breaks with MATCH

from dash_extensions.enrich import callback, Output, Input, html, DashProxy, LogTransform, DashLogger, MATCH, ALL
app = DashProxy(__name__, compress=True, transforms=[LogTransform()], prevent_initial_callbacks=True)

app.layout = html.Div(
   [
       html.Button("A", {"type": "button", "index": "A"}),
       html.Button("B", {"type": "button", "index": "B"}),

       html.Div(id={"type": "log-output", "index": "A"}),
       html.Div(id={"type": "log-output", "index": "B"}),
       html.Div(id="log-output-all"),
   ]
)

@callback(
   Output({"type": "log-output", "index": MATCH}, "children"),
   Input({"type": "button", "index": MATCH}, "n_clicks"), log=True
)
def this_fails(clicks, dash_logger: DashLogger):
   """
   log=True, dash_logger doesn't support MATCH. It supports ALL though
   """
   dash_logger.info('hello MATCH')
   return 'hello MATCH'

@callback(
   Output("log-output-all", "children"),
   Input({"type": "button", "index": ALL}, "n_clicks"), log=True
)
def this_works_if_you_remove_the_func_above(clicks, dash_logger: DashLogger):
   dash_logger.info('hello ALL')
   return 'hello ALL'

if __name__ == "__main__":
   app.run_server(debug=True)

"""
Mismatched `MATCH` wildcards across `Output`s
In the callback for output(s):
 {"index":MATCH,"type":"log-output"}.children
 notifications_provider.children@e38dd92f7bf6b8a5b59f96ecbdae5bea
Output 1 (notifications_provider.children@e38dd92f7bf6b8a5b59f96ecbdae5bea)
does not have MATCH wildcards on the same keys as
Output 0 ({"index":MATCH,"type":"log-output"}.children).
MATCH wildcards must be on the same keys for all Outputs.
ALL wildcards need not match, only MATCH.
"""

Thanks for a good, clear example! Indeed it does now work. However, I wouldn’t consider this a bug, but rather expected behaviour. I have added a note to the docs to make this more clear.

This reason why is doesn’t work is that Dash doesn’t support a combination of outputs using MATCH (i.e. the outputs you have specified) and outputs that do not use MATCH (i.e. the output that the log transforms adds). If this limitation was lifted, I believe it should work.

One way to work around this limitation, and to add support for background callbacks, would be to decouple the logging mechanism from the callback itself. I am considering adding a new transform that does this, but I am hesitant to do so as this approach would require a much more complicated architecture. First, we would need “somewhere” to collect logs in transit (i.e. a backend similar to what is used in the serverside output transform), and secondly, we would need a mechanism for pulling the logs into Dash (e.g. via an interval component, or through a web socket). Hence the resulting solution would be less plug-and-play as seen from a user perspective as compared to the current approach.