Slow data update in plotly js with live data

[frank]

Hello guys,

I using plotly to stream some data acquired from sensors. The application basically works as follows:

I have 4 channels from which I’m getting data. The application gets 4 traces from the server roughly every second. Each trace, contains more or less 80 samples with their respective timestamp.
On the client side I put the data in a buffer asI get it. Since the data is streamed as an array of raw bytes, each time I gets a trace I have to organize the data in a more useful way.
When I have collected all the 4 traces, I update the graph, but in order to do so, I have to take the data and organize it according to its x and y ranges and perform some calculations on the data to gets the size of the window on the buffer (This part is necessary because the user can place a time margin on the x axis values, so for instance he can select a window of as big as 3seconds).

Once I finish with the calculations, and I have my data ready and I update the graph using this style: Plotly.newPlot(‘sine-graph’, data, layout, {modeBarButtonsToRemove: buttons_load, displaylogo: false});
Everything seem to work quite fine for slow data gathering, but when I speed up the acquisition, the refreshing gets very slow or the browser gets stuck. The CPU however, never gets very high, it stays around 20/30%.Nevertheless, the browser does not support the load. Is this a good style to use the library or should I consider another way to update the graph?

Thank you.

[etienne]

Using Plotly.update should help a little. Plotly.newPlot is indeed the slowest thing you can use. If you’re just streaming data, Plotly.extendTraces might work even better, but it won’t come as a miracle solution yet. We’re planning on improving performance in our update routines in the next few months.

[frank]

Hello etienne,
thanks for your reply. The update function is improving the situation a bit indeed. However, it seems that for what I have tested so far, the bottleneck is in the time taken for the visualization or update call… I made this attempt based on this scenario: When I use a 2 seconds window, I have roughly 2000 points per trace to put in the graph with 4 traces at most. This amount is too much and chrome gets stuck. Trimming down the values, therefore forcing each trace to have not more than 100 points helps a lot, and the application does not crash. This could be a solution but it potentially causes a loss in the information. I was wondering if there is a sort of callback function or value, which gets set after the graph has been updated so that I would know when the function has finished.
Thanks

[Bruce_Sherwood]

See the event documentation at

Event Handlers

and scroll down to “Afterplot Event”, where you’ll find this information:

The event handler: plotly_afterplot, can be used to trigger an event each time a chart is plotted. This also includes re-plotting after the restyling or relayout of a plot. Users also have the option of adding a post-plot handler to the plot call with the following syntax: Plotly.plot(graphDiv, data, layout, config).then(postPlotHandler);

I do the following, where in other parts of my project, if lock is true, I store up data to be plotted later, once the lock is set to false by completion of the plot:

this.__lock = true
var self = this
Plotly.extendTraces(this.__id, { x:xs, y:ys }, indices).then(function() {
self.__lock = false
})

[frank]

Thanks Bruce, you pointed me towards the right direction.

[mkarhade]

Have you thought about dynamically thinning the data?
I mean there is no reason to plot more than the points/pixels on the screen at any given time in point.

So just thin out timeseries using number of points in the trace and number of pixels available in on the screen
n = len(trace) / pixels_for_image

data_to_plot = whole_data[::n]

This should give you a manageable data to plot no matter how many points you have in the trace.