Multiple chained callbacks to modify data based on user input avoiding loading from server

Dash Python
1

I’m building an app which displays a 3D scatter of some points queried from an API. The API is reaaaallly slow (average response time 15 seconds) so it is really important to avoid useless queries to the server if it’s not really necessary.

I need to pull the data based on a user input (dropdown) and then update the data, without running the query again, using a slider to translate all the points in x, y or z. The first part is easy to do: I have a function to query the dataset, query_dataset, which returns a dataframe. This uses the value of the dropdown through a callback

@app.callback(
    Output('3d-plot', 'figure'),
    [Input('msn-dropdown', 'value')])
def update_figure(msn):
    df = query_dataset(msn=msn)

    fig = go.Figure(data=[create_trace(df, 'Port', name='Port'),
                    create_trace(df, 'Stbd', name='Stbd')], layout=figure_layout())
    return fig

This works pretty well although the user has to wait about 20 seconds as I have to parse the list of all the options for the dropdown and the data itself, and this needs to be done in 2 distinct queries.

Now that the data is loaded I want to be able to move all the points doing a simple translation in x, y or z based on 3 sliders input. I can define a function to do that

def translate_df(df, dx=0):
   # dx here would be the value of a slider
   df.loc[:,"X"]+= dx
   return df

Then I would insert this into update_figure after querying the dataset. Note that I cannot do this transformation inside update_figure (this is what I’ve found as answer in many questions on this forum) as I cannot run the query again and again whenever I do a transformation.

I thought about different strategies but I couldn’t really figure it out how to do this. I was thinking of defining a callback for translate_df which would respond to the input of the sliders but then I don’t know how to pass df.

Does anyone have any idea?

2

The key to solving you issue is to cache the data after it has been queried. You can then load the data from the cache each time you need to do a transformation (based on the slider value). Since the transformation of the df is in python, it will happen server side. Hence, if the data amount is large, it would probably be most effective to save the data serverside, e.g. in a file. Here is a conceptual illustration of how it could be done,

import os
import pandas as pd
import tempfile
import dash_core_components as dcc

...
dcc.Store(id="store")  # data store for storing tmp file name
...

@app.callback(Output('store', 'data'), [Input('msn-dropdown', 'value')])
def update_dataset(msn):
    # Query the data (expensive operation).
    df = query_dataset(msn)
    # Write the data to a (unique) file (server side).
    tmp_file = os.path.join(tempfile.gettempdir(), pd.util.hash_pandas_object(df))
    df.to_pickle(tmp_file)
    # Save the file name in a data store (client side).
    return tmp_file


@app.callback(Output('3d-plot', 'figure'), [Input('store', 'data'), Input('slider', 'value')])
def update_figure(tmp_file, slider_value):
    # Load data from file.
    df = pd.read_pickle(tmp_file)
    # Manipulate the data based on slider value.
    df = translate_df(df, slider_value)
    # Create the plot.
    return make_3d_plot_from_df(df)

Next time, please post a working MWE (i.e. a piece of code that can run) which demonstrates the issue :slight_smile:

3

hey Emil, that makes sense! I was only missing the fact that you can use the type ‘data’ to store the output of a callback!
I just discovered flask-caching and @memoize today…would that have the same ‘effect’?

I didn’t post a MWE because it would have been too long and clutter the discussion. I preferred to keep it simple & short. Let me just try this solution and then I will post the full code.

5

Here is my final MWE. I even discovered uirevision to keep the figure with the same camera when the data is updated, really cool!

import dash
import dash_core_components as dcc
import dash_html_components as html
from dash.dependencies import Input, Output
import plotly.graph_objects as go
import tempfile
import os
import pandas as pd
from utils import *

external_stylesheets = ['https://codepen.io/chriddyp/pen/bWLwgP.css']

msn_list = query_msn_list()
dropdown_options = []
for msn in msn_list:
    dropdown_options.append({"label": msn, "value": msn})

app = dash.Dash(__name__, external_stylesheets=external_stylesheets)
server = app.server

app.layout = html.Div([
    dcc.Dropdown(
        id='msn-dropdown',
        options=dropdown_options,
        value=10114
    ),
    dcc.Slider( id='slider-x', min=-5000, max=5000, step=100, value=0,
                     tooltip = { 'always_visible': True }),
    dcc.Graph(id='3d-plot'),
    dcc.Store(id="store")
])

@app.callback(Output('store', 'data'), [Input('msn-dropdown', 'value')])
def update_dataset(msn):
    # Query the data (expensive operation).
    df_maa = query_dataset_maa(msn=msn)
    df_bro = query_dataset_bro(msn=msn)
    # Write the data to a (unique) file (server side).
    tmp_file_maa = os.path.join(tempfile.gettempdir(), pd.util.hash_pandas_object(df_maa)[0].astype(str))
    tmp_file_bro = os.path.join(tempfile.gettempdir(), pd.util.hash_pandas_object(df_bro)[0].astype(str))
    df_maa.to_pickle(tmp_file_maa)
    df_bro.to_pickle(tmp_file_bro)
    # Save the file name in a data store (client side).
    return {'tmp_file_maa':tmp_file_maa, 'tmp_file_bro':tmp_file_bro}

@app.callback(
    Output('3d-plot', 'figure'),
    [Input('store', 'data'), Input('slider-x', 'value'), Input('msn-dropdown', 'value')])
def update_figure(tmp_file, slider_value, msn):
    # filtering the data
    df_maa = pd.read_pickle(tmp_file['tmp_file_maa'])
    df_bro = pd.read_pickle(tmp_file['tmp_file_bro'])

    df_bro = translate_df(df_bro, slider_value)
    layout_fig=figure_layout() 
    # Add reference to MSN to avoid reloading of UI
    layout_fig['uirevision'] = msn
    fig = go.Figure(data=[create_trace(df_maa, side='Left', name='MAA Left'),
                          create_trace(df_maa, side='Right', name='MAA Right'),
                          create_trace(df_bro, side='Port', name='BRO Left', color="red"),
                          create_trace(df_bro, side='Stbd', name='BRO Right', color="red")],
                          layout=layout_fig)
    return fig


if __name__ == '__main__':
    app.run_server(debug=False)
1 Like