Black Lives Matter. Please consider donating to Black Girls Code today.
Dash HoloViews is now available! Check out the docs.

Extend Traces without sending whole figure

Hey Dash Com,
I created an app which reads in csv files, creates a json from a few lines and pushes it into a hidden input text on my app. After the data is pushed, a custom javascript renders the data by Plotly.extendTraces. When this is done the app signals the server to send new data via another hidden input text. This runs pretty smoothly the only issues I have is, if the csv data run out and I need to wait on new data (it’s a file io stream fed by a neural network training process) and if the dom-elements of my input texts load too slowly. Any improvement would be appreciated.

Layout and Loading JS:

APP = dash.Dash(__name__, static_folder=os.path.join(os.getcwd(), 'static'))
EXTERNAL_JS = ['static/custom.js']
for js in EXTERNAL_JS:
    APP.scripts.append_script({'external_url': js})

                                          .
                                          .
                                          .

APP.layout = html.Div(children=[
                      html.Div(id='heading', className='row', style={'width': '95%',
                                                                     'margin-left': 10,
                                                                     'margin_right': 10,
                                                                     'max-width': 1920},
                               children=[
                               html.H2('HIWINDnn Training Progress'),
                      ]),
                      html.Div(className='container-fluid', style={'width': '95%',
                                                                   'margin-left': 10,
                                                                   'margin_right': 10,
                                                                   'max-width': 1920},
                               children=[
                               html.Div(className='row',
                                        children=[
                                        html.Div(className='col-md-12 col-lg-6',
                                                 children=[
                                                 dcc.Graph(id='live_sse_graph',
                                                           animate=False, figure=SSE_FIG)
                                        ]),
                                        html.Div(className='col-md-12 col-lg-6',
                                                 children=[
                                                 dcc.Graph(id='live_ae_graph',
                                                           animate=False, figure=AE_FIG)
                                        ])
                               ])
                      ]),
                      html.Div(style={'display': 'none'},
                               children=[dcc.Input(id='sse_cache', type='text', value=''),
                                         dcc.Input(id='sse_rendered', type='text', value='')]),
                      html.Div(style={'display': 'none'},
                               children=[dcc.Input(id='ae_cache', type='text', value=''),
                                         dcc.Input(id='ae_rendered', type='text', value='')])
             ])

Callback:

    @APP.callback(Output('sse_cache', 'value'),
                  inputs=[Input('sse_rendered', 'value')])
    def update_sse(flag):
        """
        Pushes data into a hidden div to be rendered as for the SSE Graph
        TODO: Wow this is multithreaded....
        """
        if flag == 'restart':
            sse_csv_s.seek(0)
        if flag == '':
            raise PreventUpdate('Client Cache not done processing')

        data = {'epoch': [],
                'loss_train': [],
                'loss_val': [],
                'metric_train': [],
                'metric_val': []}        
        while len(data['epoch']) <= 0:
            for sselog in sse_csv_s.readlines(400):
                if not (sselog == '' or  sselog.startswith('epoch')):
                    logdata = list(map(float, re.split(r'\t+', sselog.rstrip('\n'))))
                    data['epoch'].append(logdata[0])
                    data['loss_train'].append(logdata[1])
                    data['loss_val'].append(logdata[2])
                    data['metric_train'].append(1-logdata[3])
                    data['metric_val'].append(1-logdata[4])
            time.sleep(0.5)
        return json.dumps(data)

JS:

function set_input(element, val) {
    element.value = val;
    var event = new Event('input', {
        'isTrused'   : true,
        'bubbles'    : true,
        'cancelable' : false
    });
    element.dispatchEvent(event);
}

function update_train_graph(cache, graph_id) {
    console.log('Plotting');
    if (cache.value) {
        data = JSON.parse(cache.value);
        for (i = 0; i < data.epoch.length; i++) {
            var x = [[data.epoch[i]], [data.epoch[i]], [data.epoch[i]], [data.epoch[i]]]
            var y = [[data.loss_train[i]], [data.loss_val[i]], [data.metric_train[i]], [data.metric_val[i]]]
            Plotly.extendTraces(graph_id, {x: x, y: y}, [0, 1, 2, 3]);
        }
    }    
}

function update_ui(cache, flag, graph_id) {
    var cache = document.getElementById(cache);
    var flag = document.getElementById(flag);
    set_input(flag, '');
    update_train_graph(cache, graph_id);
    set_input(flag, 'ready');
}

// Need more reliable wait for dom, THANKS react!
document.onreadystatechange = function () {
  if (document.readyState === "complete") {
    //Plot 1
    var observer = new MutationObserver(function(mutations) {
        mutations.forEach(function(mutation) {
            update_ui('sse_cache', 'sse_rendered', 'live_sse_graph');
        });
    });
    var cache = document.querySelector('#sse_cache');
    observer.observe(cache, { attributes: true, childList: true });
    set_input(document.querySelector('#sse_rendered'), 'restart')

    //Plot 2
    var observer = new MutationObserver(function (mutations) {
    mutations.forEach(function (mutation) {
            update_ui('ae_cache', 'ae_rendered', 'live_ae_graph');
        });
    });
    var cache = document.querySelector('#ae_cache');
    observer.observe(cache, { attributes: true, childList: true });
    set_input(document.querySelector('#ae_rendered'), 'restart')
  }
}

Even if none in the community has any suggestions for improvements, I thought I’d share it

1 Like

Wow, this is a pretty cool solution, thanks for sharing!

The “Dash” way to do this would be to create a custom React component that manages this in the Dash life cycle. There might be a property like extend_data that, when updated, would call extendTraces.

We’re working on some better docs for creating React components from first principles here: https://github.com/plotly/dash-docs/pull/116

I’ve forgot to mention. The definitions of my callbacks are in another method’s scope which opened the filestream. As I don’t know their location before runtime. Bit hacky but it works :smiley:
That’s why I reset to the file beginning instead of opening a filestream everytime I get a callback.