Real-Time Heatmap Streaming with dcc.Interval and extendData

I have seen multiple demos to show how to use extendData for streaming data in real-time, but all the examples were with scatterplots. I’m not sure if it would work with heatmap.

I want to stream a heatmap in real-time from local text files on my computer. These files are generated by software every 5 seconds. So, my app will check this folder to find which file was added to the folder. Then it would load the file and assign the new values for xnew, ynew, and znew.
xnew is the timestamp, ynew and znew are other physical properties at each timestamp. Then return [dict(x=[xnew], y=[ynew], z=[znew])] to “extendData” of the figure.

It appeared that the app was able to load the first and second files because I can see a change in the x-axis. However, as the third file was generated in the folder, it seems that the heatmap did only update the z values but not the x-axis. It looked like the heatmap was stuck at the same timestamp. I have printed out the new, ynew, and znew values and confirmed that the app was able to find the newly generated file and load it. But I’m not sure why the heatmap did not update the x-axis.

I’m not sure what I have done incorrectly here. Have any of you done this before?
@Emil, @Emmanuelle, @brad would you advise me on this issue, please?

figure=dict( data=[{'x': [], 'y': [],'z':[], 'type':'heatmap', 'colorscale':jet_scale, 'zmin':-5, 'zmax':5, 'zsmooth':'best'}],
        layout=dict(width=1900, height=1000, xaxis1=dict(domain=[0,0.97], anchor='y1', ),
            yaxis=dict(titile='<b> Depth (ft)</b>', range=[16000,9000])))

#%%%
'''
    Define Dash Layout
'''
# app = dash.Dash(__name__) 
app=Dash(__name__, prevent_initial_callbacks=True)

app.layout = html.Div( 
    [ 
        dcc.Graph(id = 'live-graph', 
                  # animate = True, 
                  figure=dict(figure)),
             
                   
        dcc.Interval( 
            id = 'interval', 
            interval = 10*1000, # 10sec
            n_intervals = 0
        ), 
        
        dcc.Store(  
        id='data-storage', data={}
        ),
        
        html.Div(id='hidden-div'),
        
        
    ] 
) 


@app.callback(
                [   
                    Output('live-graph', 'extendData')],
                [
                    Input('interval','n_intervals'), 
                 ]
    
             )
def update_data(n_interval):

    xnew=[]
    ynew=[]
    znew=[]    
    
    path=r'E:\Hess\LAS\Test'
    fn=os.listdir(path)
    
    if len(fn)==0:
        return [dict(x=xnew, y=ynew, z=znew)]
    
    else:
    
        idx=int(n_interval-1)
        
        print(f'length of fn is {len(fn)}')
        print(f'The value of idx is {idx}')

        f=fn[idx]
    
        full_path=os.path.join(path,f)
      
        new_data=function_to_load_text_file(full_path)   
        print(f'new data is {new_data}')    
    
        xnew=new_data.columns
        ynew=new_data.index
        znew=new_data.values
        
        print(f'xnew is {xnew}')      
    
        print(f'ynew is {ynew}')  
    
        print(f'znew is {znew}')      
        
        
        return [dict(x=[xnew], y=[ynew], z=[znew])]

Could you adopt the code to a runnable MWE, including generation of dummy data demonstrating the issue?

Thanks @Emil for the quick reply.

For clarification, I have attached my scripts and files, so you can check it out just in case.

There are 2 scripts: helper_function.py and Real Time Heatmap.py. The first one is to move the file from Folder 1 to Folder 2. The latter script is to check Folder 2 every 10 seconds, find the new generated file and load it to the app in order to update it in real-time.

There is only one folder in the attachment: “Folder 1”. This folder contains the files and serves as the source folder. You will need to create another folder “Folder 2”, so this folder will be used as the destination folder.

To run the app:
1/ Create “Folder 2” on your computer
2/ Update the directory for “Folder 1” and “Folder 2” in line 8 and 11 of “helper_function.py” and line 18 of “Real Time Heatmap.py”
3/ I have to use 2 instances of Spyder in order to run my app. Run the first instance of Spyder with “Real Time Heatmap.py” to get the app start. Then run a second instance of Spyder for “helper_function.py”

After you run it, you will see that the last timestamp should be 01:04:50 (because it is the last file in the folder). However, the figure displayed as 01:04:00 instead.

I’m still not sure what I have done wrong here, and I would really appreciate your time and effort to help me to figure it out.

And the files are attached here
https://drive.google.com/file/d/1A0hmuRc8jPA61oILGfEY5tMNEKFTjTQX/view?usp=sharing

Does anyone have any idea what went wrong here?
Thanks