I’m working on a Dash app which gets data from a server and treats the data within the app. I want to check if there is data from the server by a callback (“Server_data”), periodically triggered by the dcc.interval component.
If there is new data, I store this data into a dcc.store and raise a “data_flag”.
I want to trigger an other callback (“handling_data”) by that dcc.store to update my graphs, plots, tables etc. only if there is new data from the server.
I switch the “data_flag” back to False as soon as I’m done with the data processing within the “handling_data” callback, just to allow the “Server_data” callback to get new data from the server again.
Everything works fine so far as long as the dcc.intertval’s interval is not too short. Too short means in the order of 100 ms. If I set it to such a small intervall, it fails to trigger the “handling_data” callback sometimes. This causes the “data_flag” to stay True and I will not get any data from the server anymore.
It is getting worse as shorter the interval becomes.
I’m now afraid that this interval “threshold” depends on the PC or Laptop I’m using.
Is there any fundamental time limitation for the dcc.interval or for triggering a dash callback?
Here is my code example…if you change the interval to, for example, 10 ms, you can encounter the problem rather fast.
I’m using Python 3.10 with Dash version 2.12.1
Thank you already for your support!!
from os import times
import time
import dash
from dash import State, dcc, html
from dash.dash import PreventUpdate
from dash.dependencies import Input, Output
import pandas as pd
import datetime
# define a class to init the data Flag and the Server count
class Flag:
def __init__(self):
self.data_Flag = False
self.Server_count = 0
# create an instance of the Flag class
Flag_class = Flag()
app = dash.Dash(__name__)
app.layout = html.Div([
dcc.Interval(
id='interval-component',
interval=50, # The "handling_data" callback is not triggered at some point if this intervall gets too short
n_intervals=0
),
dcc.Store(id='data-store', storage_type = 'session', data={'time': [], 'value': []}),
html.Div(id='data-display'),
])
@app.callback(
Output('data-store', 'data'),
[Input('interval-component', 'n_intervals')]
)
def Server_data(n_intervals):
# check if the "display_data" function is done with the data comming from the Server.
if Flag_class.data_Flag == True:
print("Data Flag True")
raise PreventUpdate
else:
# getting new data from the server
print("Entered Interval")
# count how often we entered this case
Flag_class.Server_count = Flag_class.Server_count+1
# set the data flag to True to prevent getting data from the serevr as long as handling data is not done
Flag_class.data_Flag = True
# empty the data to avoid an overflow of the dcc.store
data = {'time': [], 'value': []}
# Simulate data update
current_time = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
new_value = Flag_class.Server_count
data['time'].append(current_time)
data['value'].append(new_value)
print("Exit Interval")
return data
@app.callback(
Output('data-display', 'children'),
[State('data-store', 'data'),
Input('data-store', 'modified_timestamp')], prevent_initial_call = True
)
def handling_data(data, timestamp):
# this callback should always be called if the dcc.store changed
# it works most of the time, but if the dcc.interval is too short, it sometimes does not trigger this callback
# put the data int oa dataframe
df = pd.DataFrame(data)
print(f"Dataframe length = {len(df)}")
time.sleep(1)
# set thedata flag to False to "allow" the Server_data to get new data from the server
Flag_class.data_Flag = False
print(f"Data Flag {Flag_class.data_Flag}")
# return the data to the output
return html.Table(
# Header
[html.Tr([html.Th(col) for col in df.columns])] +
# Body
[html.Tr([html.Td(df.iloc[i][col]) for col in df.columns]) for i in range(len(df))]
)
if __name__ == '__main__':
app.run_server(debug=True)
Working with a flag like this will only work if you have a single user simultaneously and a single server. If multiple workers are involved, each worker will have their own copy of the code and thus their own instance of the Flag class. If all workers are allowed to process callbacks coming from any user, then the state of the Flag class will not be what you expect it to be (even if you only have 1 user). Furthermore, the app is no longer stateless, and actions of one user will affect the results of other users. So, in the context of Dash apps, using the Flag class is bad design.
Why not increase the polling time? Since you are skipping most of the updates, why not just do less updates? It’s unclear to me exactly why the second callback fails updating, but what you could try to do is check in the browser developer tools if the timestamp of the store is actually changes. Only when that number changes will the callback execute.
One other approach you can do is to use the data property as input: Input('data-store', 'data'). You need the modified_timestamp only when you need to fetch the data when loading the dashboard. However, as you set prevent_initial_call=True, you clearly don’t need that. This does have slightly different behavior. I have seen sometimes that the modified timestamp did not change, even though I wrote data to a store, but when I had the data as Input it would still trigger the callback. Note here that Dash has some internal smartness where if you push exactly the same data twice to the store, it will not count as an update. So if the store currently holds the value 3 and in a second execution of the callback you again write 3 to the store, it won’t trigger the callback. (but maybe that is only the case when using the modified_timestamp property to trigger the callback.
Lastly, note that callbacks are triggered independently, irrespective of whether the others are running or not. That is, if the interval is triggered faster than the interval callback needs to fully execute, multiple copies of the first callback will run. Potentially, you run in a situation with some race condition and update come in too fast and either two runs of same callback result in some interference or blocks the browser from correctly registering that the store has changed.
thank you for the fast response!
I’m planing to only have one user at a time but the comment with the Flag class is very helpful anyways!
I want to have a rather short polling time as I don’t know when the server sends some data. I want to avoid an overflow of my queue in which the server stores the data.
Anyways, I tried a polling time of 500 ms over night and this worked without any error.
I also used the ‘data-store’, ‘data’ as an input but the result was the same.
I’m still wondering why the second callback is not triggered some times and if the “critical polling time” depends on the PC or Laptop which is used to run the app.
I have done some experimentation with your dashboard. And my conclusion is that you are swamping the server with requests when the interval is too low, and the server simply cannot keep up. I can get it to work with an interval of 30 ms, however one hickup in the browser or server and the updates get stuck. Furthermore, due to the high rate the interval component triggers, the second callback will always run twice simultaneously. This is because the flag is not toggled fast enough. My recommendation is that you rethink your approach. The following suggestion comes to my mind:
Instead of using the flag, just toggle the disabled property of the interval component. Put the interval at bit higher, like 100ms so that all callbacks have time to execute and the browser has time to update all components. Then make sure that the first callback disables the interval when exiting that callback. The second callback can then enable the interval again after it has finished updating the data. This removes the need for the Flag class, makes your app stateless again, and reduces the strain the app exerts on the server.
Note: This approach need duplicate callback outputs. To enable this feature, set allow_duplicate=True, check duplicate callback outputs docs for more info.
If you really do want to work with global states, you can make a dcc.Store that holds the global value. You can pass the data of the Store as a State into the callback. This will again make the application stateless. But in your case, I think it’s cleaner to disable the interval.
Toggling the disabled property of the interval component sounds like a very good idea!
Thank you! I should avoid unnecessary triggering of any callbacks with this…!
I’ll try it like this and see if this casues any other issues.