Announcing Dash Bio 1.0.0 🎉 : a one-stop-shop for bioinformatics and drug development visualizations.

Streaming sensor data at >100 points per second

We’ve been unable to get plotly-dash to smoothly display a constant stream of data.

A few examples we’ve been following so far all use Interval to update the graph:

But in our experience with higher data rates the interval callback can take too long to process. When the interval timer comes due it then re-enters the callback for a second time (while the first callback is processing), and when this happens plotly basically crashes, nothing renders. Using print statements, we can see that the callbacks get backed up and don’t run at smooth intervals.

I made an attempt to work around the re-entrant callback problem using an interval callback which either triggers the graphing callback using a dcc.Store update or raises PreventUpdate depending on whether the last update ran. But I got quirky results where the dcc.Store values, which I used to indicate that the graphing callback completed, weren’t getting updated or weren’t visible to the interval callback.

That approach seems like a hack anyway.

I’m looking for ideas on what to try next. Is there hope for Plotly/Dash to be able to stream 100+ data points per second smoothly? Or perhaps plotly just isn’t made for this use case?

What refresh rate do you need to get what you would consider a smooth update? 5s? 1s? 0.1s?

0.1s is 10 frames per second, that’s generally about the minimum refresh rate that looks smooth to a human. I only want to show 5 seconds of data, so ~500 points to plot as the data scrolls by. If the update happened every second then 100 new data points would show up, so 1/5th of the graph, and when we do that it’s clearly choppy.

I was hoping that a single render of say 10 seconds of data with quick updates to a range property would be possible, but that doesn’t seem like an option.

I also noticed that Plotly V3 had a Streaming API (https://plotly.com/python/v3/streaming-tutorial/) which looks closer to what is needed to achieve this, but as far as I can tell that doesn’t exist in V4, only the interval callbacks which are quite limited at higher frame rates needed to smoothly stream data.

There’s also this old streaming demos plotly repo that hasn’t been updated in 2 years and all the demos don’t work now. But there’s a video showing a streaming voltage trace which is exactly what we’re trying to accomplish.

Have you tried using extendData? that’s the state of the art for dash at the moment

I have not used extendData yet, this sounds promising, but I’m not sure I understand how to apply it.

In the plotly dash docs (https://dash.plotly.com/dash-core-components/graph) I see that extendData is a property of the graph object. Does the callback look like this:

@app.callback(
  output=Output('my-graph', 'extendData'),
  inputs=[Input('interval-component', 'n_intervals')])
def extend_graph(n_intervals):
  return [get_new_data(), get_trace_indices(), get_max_points()]

I just did a few tests. With normal callbacks, i can get down to around 100 ms (which is not perfectly smooth). With clientside callbacks, i can get to updates around 10 ms, which looks smooth to be. I posted an example on the use of extendData in your stack overflow thread.

1 Like

Oh brilliant, thank you so much for that. We can incorporate the client side callback once we get the server-side method working. 100ms is plenty good enough for a quick proof of concept.

And here is a giff using the clientside update mechanism :slight_smile: . I guess with client side updates, the data flow would be,

  • Stream the data (server->client) in chunks into a Store component every second (or 5s or whatever)
  • Stream the data (client->client) smoothly from the Store to the Graph every 10 ms (or maybe just 50 ms)

Note that the 100 ms i mentioned before was for my setup, it may vary depending on network speed, hardware and update complexity. In general, for anything faster than 1s refresh rate, i would prefer client side updates.

3 Likes

Beautiful example @Emil! Thanks for digging in here, cool strategy with slower store updates coupled with faster clientside updates

Thanks, @chriddyp. I have now updated my answer on stackoverflow with the exmple code :slight_smile: