Load and store new dataset or model on callback

Dash Python
1

Hi everyone in the Dash community!
I’m new to Dash, so I apologize if my question seems silly, but I haven’t found any solution yet.

I’m doing research in AI and I’m creating an app to make some demos of what I’m building. To do so, I’d like to able to load several datasets I store locally (my app is not meant to be hosted so far) from a dmc.Select menu. When a dataset is selected, it will call some callbacks to update other dmc.Select menus that will update some figures.
I was thinking about using dcc.Store as advised in the docs, but my datasets are quite heavy (almost 100MB each). Also, I want to further be able to load some pretrained models that could be selected from an other dmc.Select menu, which raises, I think, the same question.

Here is a glimpse of my code so that you can get a better idea. I call some functions defined in other scripts, but I think their output is rather straightforward to understand. Still, if you have any question, don’t hesitate to ask me to clarify whatever you need!

I’m aware that I’m not using dcc.Store in the right way because I’m not creating a json file, but I leave it here so you can get the idea (I hope).

Thank you all for your help!

import dash_mantine_components as dmc
from dash import Dash, dcc, html, Input, Output, State

from helpers import *

import json
import plotly.graph_objects as go

data_dir_list = get_avaible_datasets_list()
load_function_dict = {
    "kpi": load_kpi_data  # ,
    # "smd": load_smd_data,
    # "nasa": load_nasa_data,
}
model_dropdowm = get_avaible_models_list()

app = Dash(__name__)
app.config.suppress_callback_exceptions = True

app.layout = html.Div(
    [
        html.H1("Counterfactualizer"),
        html.Div(
            [
                dmc.Select(
                    label="Select dataset (Only KPI available for now)",
                    data=data_dir_list,
                    value=data_dir_list[0]["value"],
                    id="dataset-dropdown",
                ),
                dcc.Store(id = "dataset", "data"),
                html.Div(children=[], id="signal-dropdown"),
            ]
        ),
        html.Div(children=[], id="ts-plot"),
        html.Div(
            [
                dmc.Select(
                    label="Select model",
                    data=model_dropdowm,
                    id="model-dropdown",
                    value=model_dropdowm[0]["value"],
                )
            ]
        ),
        html.Div(children=[], id="detection-plot"),
    ]
)


@app.callback(Output("dataset", "data"), Input("dataset-dropdown", "value"))
def update_dataset(dataset_name):
    load_function = load_function_dict[dataset_name]
    train_set, test_set = load_function(f"{data_dir}/{dataset_name}")
    global test_set


# update signal-dropdown when dataset is changed
@app.callback(Output("signal-dropdown", "children"), Input("dataset", "data"))
def update_signal_dropdown(test_set):
    id_list = get_signal_list(test_set)
    return dmc.Select(label="Select signal", data=id_list, value=id_list[0]["value"])


@app.callback(
    Output("ts-plot", "children"),
    Input("signal-dropdown", "value"),
    State("dataset", "data"),
)
def update_figure(signal, dataset):
    return dcc.Graph(
        figure=create_timeseries_from_dataset_and_id(dataset, signal).plot_plotly(),
    )


@app.callback(
    Output("detection-plot", "children"),
    Input("signal-dropdown", "value"),
    State("dataset", "data"),
)
def update_detection_figure(signal, dataset):
    model_path = "/home/paperspace/models/usad_kpi_2023-04-14_10:44:47_epochs_80_window_size_5_hidden_size_10_downsampling_0.01"

    with open(f"{model_path}/args.json", "r") as f:
        config = json.load(f)

    model = load_usad_model(model_path, config, get_default_device())

    ts_predicted = assign_labels(signal, model, dataset)
    return dcc.Graph(figure=ts_predicted.plot_plotly(is_real_labels=False))


if __name__ == "__main__":
    app.run_server(debug=True)
2

Hi @vthieb welcome to the forums.

I used the ServerSideOutput from dash-extensions for something similar in the past.

3

Hi @AIMPED !

Thank you for your reply, I didn’t know about dash-extensions. Unfortunately, it does not solve my problem because it raises a Duplicate callback outputs error that I don’t understand because each output in a callback is created only once.
I also tried to run the simple example from your link and I get the same error. Any idea of what could be going on?

4

Could you share your code (the one where you are using dash-extensions)?

5

Sure, here you go.
As mentioned in my previous answer, the example from your link does not work either. Is it also the case for you?

import dash_mantine_components as dmc
from dash_extensions.enrich import (
    DashProxy,
    Output,
    Input,
    State,
    ServersideOutput,
    html,
    dcc,
    ServersideOutputTransform,
)
from helpers import *

import json

data_dir_list = get_avaible_datasets_list()
load_function_dict = {
    "kpi": load_kpi_data  # ,
    # "smd": load_smd_data,
    # "nasa": load_nasa_data,
}
model_dropdowm = get_avaible_models_list()

app = DashProxy(transforms=[ServersideOutputTransform()])

app.layout = html.Div(
    [
        html.H1("Counterfactualizer"),
        html.Div(
            [
                dmc.Select(
                    label="Select dataset (Only KPI available for now)",
                    data=data_dir_list,
                    value=data_dir_list[0]["value"],
                    id="dataset-dropdown",
                ),
                html.Button("Load dataset", id="load-dataset"),
                dcc.Store(id="dataset"),
                html.Div(children=[], id="signal-dropdown"),
            ]
        ),
        html.Div(children=[], id="ts-plot"),
        html.Div(
            [
                dmc.Select(
                    label="Select model",
                    data=model_dropdowm,
                    id="model-dropdown",
                    value=model_dropdowm[0]["value"],
                )
            ]
        ),
        html.Div(children=[], id="detection-plot"),
    ]
)


@app.callback(
    ServersideOutput("dataset", "data"),
    Input("load-dataset", "n_clicks"),
    State("dataset-dropdown", "value"),
    prevent_initial_call=True,
)
def update_dataset(n_clicks, dataset_name):
    print("update_dataset")
    load_function = load_function_dict[dataset_name]
    train_set, test_set = load_function(f"{data_dir}/{dataset_name}")
    return test_set


# update signal-dropdown when dataset is changed
@app.callback(
    Output("signal-dropdown", "children"),
    Input("dataset", "data"),
    prevent_initial_call=True,
)
def update_signal_dropdown(test_set):
    id_list = get_signal_list(test_set)
    return dmc.Select(label="Select signal", data=id_list, value=id_list[0]["value"])


@app.callback(
    Output("ts-plot", "children"),
    Input("signal-dropdown", "value"),
    State("dataset", "data"),
)
def update_figure(signal, dataset):
    return dcc.Graph(
        figure=create_timeseries_from_dataset_and_id(dataset, signal).plot_plotly(),
    )


@app.callback(
    Output("detection-plot", "children"),
    Input("signal-dropdown", "value"),
    State("dataset", "data"),
)
def update_detection_figure(signal, dataset):
    model_path = "/home/paperspace/models/usad_kpi_2023-04-14_10:44:47_epochs_80_window_size_5_hidden_size_10_downsampling_0.01"

    with open(f"{model_path}/args.json", "r") as f:
        config = json.load(f)

    model = load_usad_model(model_path, config, get_default_device())

    ts_predicted = assign_labels(signal, model, dataset)
    return dcc.Graph(figure=ts_predicted.plot_plotly(is_real_labels=False))


if __name__ == "__main__":
    app.run_server(debug=True)
6

Your code looks good, obviously I can’t tell what is happening in the functions you use.

The example from the dash-extensions page work fine, no errors on my side

7

Thanks for trying!

After a few tries, I think it simply is caused by the python version I’m using (3.6). I tried the example using 3.8.8 and it works. I’ll try to do the same with my project environment, hoping it won’t break other dependencies.

1 Like