Extend (or append) data instead of update

Dash Python
1

I´m writting a live app that loads a large amount of data into the grahp. Currently I´m using the same method as in the examples.

app = dash.Dash(__name__)

app.layout = html.Div(
    html.Div([
        html.H4('Title'),
        html.Div(id='update-text'),
        dcc.Graph(id='update-plot'),
        dcc.Interval(
            id='interval-component',
            interval=10*1000,  # in milliseconds
            n_intervals=0
        )
    ])
)

@app.callback(Output('update-plot', 'figure'),
              [Input('interval-component', 'n_intervals')])
def update_plot(n):
    # load data from db
    # create the taces
    
    return Figure(data= traces, layout = SOME_LAYOUT)

if __name__ == '__main__':
    app.run_server(debug=True)

The problem here is that every 10 seconds all the data have to be reloaded and redraw and it takes some time. Another issue is that if have some zoomed area or some trace hidden I loose it after the update.

I there any way to just add the latest data to the traces? Something as Plotly.extendTraces.
i´m gessing something like this

@app.callback(Output('update-plot', 'figure'),
              [Input('interval-component', 'n_intervals')])
def update_plot(n):
    if n==0: # First update
        # load data from db
        # create the taces
       # return the Figure
    else:
       # append new data to the Figure

Thanks

2 Likes
2

You could add a State to the callback that captures the figure property:

@app.callback(Output('update-plot', 'figure'), [Input('interval-component', 'n_intervals')], [State('update-plot', 'figure')])

This will give you access to the current value of figure, which you can get the traces data out of.

4 Likes
3

Great, this works. Now I have to figure out how to keep the same zoom after the update…

Thank you!

[EDIT] Here is solved

1 Like
4

Is there a best practice for how to use the fileopt='extend' argument? I am currently using the figure state in my callback, but maybe not in the most efficient manner.

What I did was basically something like:
x_new = fig['data'][0]['x'].extend(new_x_values)
y_new = fig['data'][0]['y'].extend(new_y_values)

and then creating a brand new Scatter trace with x=x_new, y=y_new. This works, but on the network side I’m still sending the complete dataset to the client on every interval.

5

Do it this way surely save time in retrieving old data from the database, but it still redraws everything on the plot by
return {‘data’: traces, ‘layout’: layout}

is there a way to only draw the new trace and keep existing traces on the plot without redrawing them.

Thanks

1 Like
6

There is not. You have to use the method suggested above: Extend (or append) data instead of update - #2 by nedned.

It might be nice to have some “extend” option for the dcc.Graph component so that we don’t have to send the data back and forth to the client. I haven’t really thought this through, but if you want to play around with the dcc.Graph JS source code here: dash-core-components/src/components/Graph.react.js at master · plotly/dash-core-components · GitHub

7

So this confuses me a bit. The state is the graph and figure and the interval is the trigger.
How is the state actually forcing an extension to the trace instead of reloading the trace/graph?

I have mimicked the callback but I get an error.

TypeError: update_graph_scatter() takes 0 positional arguments but 2 were given
127.0.0.1 - - [20/Oct/2018 18:49:36] “POST /_dash-update-component HTTP/1.1” 200 -

exchange_tickers = {
‘Binance’: [‘XRP/USDT’],
‘Bittrex’: [u’XRP/USDT’],
‘Kraken’: [u’XRP/USD’],
‘Poloniex’: [u’USDT_XRP’]
}
exchanges = [‘Kraken’, ‘Binance’, ‘Bittrex’]
exchanges = [dict(label=str(exchange), value=str(exchange)) for exchange in exchanges]

PullValues = bf()
AppendedValue = PullValues.Kraken()

NewDateTimes = AppendedValue.index
NewclosePrice = AppendedValue[‘close’]

app = dash.Dash(name)
app.layout = html.Div([
html.Div([
html.H1(
‘Trade Pair - Exchange’,
style={‘padding-top’:‘20’, ‘text-align’:‘center’}
),
html.Div([
html.Label(‘Exchange:’),
dcc.Dropdown(
id=‘dropdown_exchange’,
options=[{‘label’: k, ‘value’: k} for k in exchange_tickers.keys()],
value=‘Kraken’,
)
], style={
‘width’: ‘300’,
‘display’: ‘inline-block’,
‘padding-left’: ‘50’
}),
html.Div([
html.Label(‘Select ticker:’),
dcc.Dropdown(
id=‘dropdown_ticker’,
options=tickers,
value=‘XRP’,
)
], style={
‘width’:‘300’,
‘display’:‘inline-block’,
‘padding-left’:‘50’
}),

    html.Div([
        html.Label('Amount To Leverage:'),
        dcc.Input(
            id='amount_leverage',
            placeholder='50',
            type='text',
            value='50',
        )
    ], style={
        'width':'300',
        'display':'inline-block',
        'padding-left':'50'
    })
]),
html.Div([
    dcc.Graph(id='initial-graph', animate=True,
              figure={
                  'data': [
                      {'x': NewDateTimes, 'y': NewclosePrice, 'type': 'line', 'name': 'LINE XRP-USD'},
                  ],
                  'layout': {
                      'title': '4 HOUR - XRP-USD - KRAKEN',  # TITLE AT TOP OF CHART
                      'legend': {'x': 'XRPUSD'}
                  }
              }),
    dcc.Interval(id='graph-update', interval=20000)
]),

])
@app.callback(dash.dependencies.Output(‘dropdown_ticker’, ‘options’), [dash.dependencies.Input(‘dropdown_exchange’, ‘value’)])
def update_trading_pair(selected_exchange):
return [{‘label’: i, ‘value’: i} for i in exchange_tickers[selected_exchange]]

@app.callback(dash.dependencies.Output(‘dropdown_ticker’, ‘value’), [dash.dependencies.Input(‘dropdown_ticker’, ‘options’)])
def update_trading_pair(exchange_tickers):
return exchange_tickers[0][‘value’]

#@app.callback(Output(‘initial-graph’, ‘figure’), events=[Event(‘graph-update’, ‘interval’)])
@app.callback(Output(‘initial-graph’, ‘figure’), [dash.dependencies.Input(‘graph-update’, ‘interval’)], [dash.dependencies.State(‘initial-graph’, ‘figure’)])
def update_graph_scatter():
####### CANDLESTICK GRAPH GRAPH ####################
PullValues = bf()
AppendedValue = PullValues.Kraken()
NewDateTimes = AppendedValue.index

trace = Candlestick(
    x=NewDateTimes,
    open=AppendedValue['open'],
    high=AppendedValue['high'],
    low=AppendedValue['low'],
    close=AppendedValue['close'],
    showlegend=False,
    name='XRP/USD',
    legendgroup='XRP/USD',
    increasing=dict(line=dict(color=colorscale[3])),
    decreasing=dict(line=dict(color=colorscale[5]))
)
layout = Layout(
    height=480,
    margin={'l': 50, 'r': 30, 'b': 50, 't': 50},
    legend={'x': 0, 'y': 1, 'xanchor': 'left'},
    xaxis={'rangeslider': {'visible': False}}
)
return Figure(data=[trace], layout=layout)
8

Why doesn’t this work?


fileopt=‘extend’

Dash references that plotly uses same syntax. My reading this is that we should be able to embed the above command somehow into the graph.
https://dash.plot.ly/dash-core-components

Graphs

The Graph component shares the same syntax as the open-source plotly.py library. View the plotly.py docs to learn more.

9

@nathand the dcc.Graph component renders your figure data via Plotly.react(…). It doesn’t seem like the dash team plans to support other Plotly.js funcs (extend, prepend, add, delete, etc).

However, I think I have something that should work for your purposes. I forked dcc.Graph and added a new property in my component called extendData. When the extendData property is updated, it will use Plotly.extendTraces(…) to update the plot.

Just pushed some demo code onto github if you want to try it out (https://github.com/bcliang/dash-extendable-graph). I haven’t really tested it except with time-series scatter plots. If you find it useful, I might generate some tests and post to show and tell.

pip install dash-extendable-graph

You can still use the figure property in your initial render (and for things like updating graph layout). However, you can also use the new property in a callback:

@app.callback(
   Output(‘initial-graph’, ‘extendData’), 
   [Input(‘graph-update’, ‘interval’)], 
   [State(‘initial-graph’, ‘figure’)])

extendData content should be of the same form as figure.data:

  • list of dicts
  • one dict per trace (e.g. {'x': xdata, 'y': ydata, ...})

Note: I pulled the initial commit from dcc v.0.42.0… it will need to be patched any time dcc.Graph is updated.

3 Likes
10

actually, I meant dcc 0.43.0 (pretty big changes in the components between 0.42.1 and 0.43.0 – event handling removed!)

11

I sorely hope this good work of yours ends up working for me, my Dash app is coming apart at the seams.

What would be the proper way to go about pushing in all one’s data into a dcc.Graph object first (so not appending just yet, and then after the first run of the callback, have each run of the callback afterwards append to the originally outputted data?

Could I do this all in one callback, or would I have to create two separate callbacks now, one for putting in a large chunk of data first, and another for incrementally updating/changing it?

12

@Mike3 thanks for the ping. So, that’s the exact use case I’ve designed dash-extendable-graph component to serve. A single callback should work, provided the logic is the same. You might want to use a second input trigger (like dcc.Location) in addition to an Interval.

FYI, the component’s behavior is to extend-then-add-then-react:

  1. If the traces exist, it will extend them
  2. If there are traces that don’t exist, it will add them (e.g. you want to add another trace to a plot that didn’t exist before)
  3. If there are no traces, it will default to the dcc.Graph() behavior in redrawing the full figure with traces.

The usage example provided in the readme should do the trick.

Now, if you’d like to wait for something more officially supported, I have created a PR in dash-core-components that would hopefully support extending data. Based on feedback, the API is slightly different. Not sure if/when that’ll happen, but sometign to keep your eye on:

4 Likes
13

Hello, brad! After a new update of Dash, I immediately went to check if extendData was added in, and, lo and behold, Dash 0.41.0 seems to have something about it.

However, I am having some issues with the syntax, which I hope you could help me out with. Using the example you posted at the bottom of your pull request, I modified it in the hopes of getting a basic graph to

  1. Plot one point first (thereby creating the trace) using n_intervals as the only Input, and
  2. use n_intervals to keep extending said trace.

Needless to say, I am getting errors, and no amount of tinkering seems to help me out. Would you give a quick glance at my code below?

import dash
from dash.dependencies import Input, Output, State
import dash_html_components as html
import dash_core_components as dcc
import random

app = dash.Dash(name)

app.scripts.config.serve_locally = True
app.css.config.serve_locally = True

app.layout = html.Div([
html.Div([

    dcc.Graph(
        id='graph-extendable',
        figure={'layout': {'title': 'Random title',
                           'barmode':'overlay'}
              }
    ),
]),

dcc.Interval(
    id='interval-graph-update',
    interval=1000,
    n_intervals=0),

])

@app.callback(Output(‘graph-extendable’, ‘extendData’),
[Input(‘interval-graph-update’, ‘n_intervals’)]
)
def create_then_extend_single_trace(n_intervals):

return [dict(x=[[n_intervals]]), [0]]

if name == ‘main’:
app.run_server(debug=True)

1 Like
14

Thanks for checking in on this. The original pull request definitely morphed a bit based on feedback from the plotly developers. Essentially, the goal was for extendData prop to match Plotly.extendTraces() behavior. Aside from the API change (which you’ve noticed), two major changes (“limitations” vs. what’s in dash-extendable-graph, which for the moment I’m keeping updated as a result):

  • extendTraces doesn’t support create-then-extend functionality – a design decision on the plotly side. The proper usage is to call Plotly.addTraces before Plotly.extendTraces
  • extendTraces doesn’t handle multiple trace types (all traces in the function call must contain ‘x’ as a key) – there’s an issue for that

So, to get the above to render properly, you need to add a data key to your figure dict. It can be empty, but the trace needs to exist before the plotly library will extend it:

app.layout = html.Div([
    html.Div([

        dcc.Graph(
            id='graph-extendable',
            figure={'layout': {'title': 'Random title',
                               'barmode': 'overlay'},
                    'data': [{'x': []}]
                    },
        ),
    ]),

    dcc.Interval(
        id='interval-graph-update',
        interval=1000,
        n_intervals=0),
])
15

Then is there no functionality for adding new traces as they come? In that case, is there a way in which to create new traces within the same callback by utilizing extendData + some other functionality then? Obviously this worries me, since we all know Dash Output's can only be assigned to one callback at a time, and if one callback can’t create mulitple traces as needed, then :cold_sweat:

At any rate, I can only imagine such functionality will be supported sooner rather than later, with your clear reasoning and proof-of-concepts.

16

Unfortunately that’s correct. One way to jump start the discussion could be to create an issue on github. In general, I like the Plotly team’s approach to maintain consistency between dash components and the underlying Plotly.js API.

Big changes in behavior/enhancements might best be left to customized components (not meaning to plug again, but e.g. dash-extendable-graph).

1 Like
17

Hmm, in that case, I will have to profile this Dash example to see which choice is more efficient: using the now built-in, but semi-full capabilities, or using the full dash-extendable-graph.

Before I do that, though, I seem to have stumbled on yet another syntax issue, which I hope you could help with. In the following code (trying now to follow your latest syntax) I am trying to update only the 1st and 5th traces (indices 0 and 4), out of the 15 empty ones I have defined.

Why is this code not working?

import dash
from dash.dependencies import Input, Output, State
import dash_html_components as html
import dash_core_components as dcc
import random

app = dash.Dash(__name__)

app.scripts.config.serve_locally = True
app.css.config.serve_locally = True

app.layout = html.Div([
    html.Div([
        
        dcc.Graph(
            id='graph-extendable',
            figure={'layout': {'title': 'Random title',
                               'barmode':'overlay'},
                    'data': [{'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []},
                             {'x': []}]
                  }
        ),
    ]),
    
    dcc.Interval(
        id='interval-graph-update',
        interval=1000,
        n_intervals=0),
])


@app.callback(Output('graph-extendable', 'extendData'),
              [Input('interval-graph-update', 'n_intervals')]
             )
def create_then_extend_single_trace(n_intervals):
    
    return (dict(x = [
                      [n_intervals],
                      [n_intervals]
                     ],
                 
                 y = [
                      [n_intervals],
                      [n_intervals**2]
                     ]
                ),
            
            [0,4]
           
           )



if __name__ == '__main__':
    app.run_server(debug=True)
18

When you initialize the traces, you need to have all the keys defined. In this case, a scatter plot needs both x and y defined.

import dash
from dash.dependencies import Input, Output, State
import dash_html_components as html
import dash_core_components as dcc
import random

app = dash.Dash(__name__)

app.scripts.config.serve_locally = True
app.css.config.serve_locally = True

app.layout = html.Div([
    html.Div([

        dcc.Graph(
            id='graph-extendable',
            figure={'layout': {'title': 'Random title',
                               'barmode': 'overlay'},
                    'data': [{'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []},
                             {'x': [], 'y': []}, ]
                    }
        ),
    ]),

    dcc.Interval(
        id='interval-graph-update',
        interval=1000,
        n_intervals=0),
])


@app.callback(Output('graph-extendable', 'extendData'),
              [Input('interval-graph-update', 'n_intervals')]
              )
def create_then_extend_single_trace(n_intervals):

    return (dict(x=[
        [n_intervals],
        [n_intervals]
    ],

        y=[
        [n_intervals],
        [n_intervals**2]
    ]
    ),

        [0, 4]

    )


if __name__ == '__main__':
    app.run_server(debug=True)

For future reference, check the browser debug console; the Plotly.js library has some exception handling which can point you to errors as it relates to dash components that don’t run.

3 Likes
19

Ok, I will keep everything you said in mind in the future. With the javascript debug console (which I have recently learned the necessity of, because of Dash), the error messages are not always the clearest, but they certainly still help.

One last question, however: is it possible to send the name aspect to individual traces using these methods? So that, if I decide to populate 5 of those above empty, default traces, can I send unique names to them?

I could think of other miscl. information that could be sent to different traces, and, as far as I am aware, your examples have not described how to do that yet.

20

Would you care to quickly comment on my previous question, regarding how to send attributes like name to different traces?

I’ve tried putting every permutation of {name : [name1, name2, ..., name5]} into the latest code example, but none of them work.