How to update Dash application based on if new files are added onto server directory

Dash Python
1

I have a dash app currently that is reading files (csv) from a directory, converting to dataframe, and then presenting them onto my dash front-end. However, my files are loaded onto the directory everyday at 4am via an ETL process, meaning new files are uploaded onto the directory. I would like to build a dash application which presents the newest files (within the past day) without having to restart my entire dash application. I also have two dropdown inputs which update my output graphs (for filtering purposes).

For testing purposes, I currently, I have files dated from Aug 31 - Sept 6. My collectdata(date) function will throw an error if the date goes beyond Sept 6

def collectdata(date): #date is a string
#I receive data by reading in the files from the directory with currentdate 
# and return a dataframe

firstDate='2022-08-31'
df=collectData(firstDate) #call the function to initially set date = Aug 31

fig=px.scatter(df, x='Time', y='Acc_InitialERabEstabSuccRate', color='NodeId', hover_name='NodeId')
#... the rest of the figures to populate the graphs initially

app.layout = html.Div([
#all my dash components
 
  dcc.Dropdown( #dropdown 1
        id='provinces',
        options=createDropdownProvinces(df),
        value='All Provinces'
    ),
    
  dcc.Dropdown( #dropdown 2
      id='my-input',
      options=createDropdown(df)
  ),

  dcc.Interval( #interval component for invoking a refresh
        id='interval-component',
        interval=1*20000, #every 20ms update the graphs 
        n_intervals=0
    ),

   #My 9 graphs
   dcc.Graph(id='my_graph', figure=fig),


  
])

@app.callback(
    Output('my_graph', 'figure'),
    Output('my_graph2', 'figure'),
    Output('my_graph3', 'figure'),
    Output('my_graph4', 'figure'),
    Output('my_graph5', 'figure'),
    Output('my_graph6', 'figure'),
    Output('my_graph7', 'figure'),
    Output('my_graph8', 'figure'),
    [Input(component_id='interval-component', 'n_intervals'),
    Input(component_id='provinces', component_property='value'),
    Input(component_id='my-input', component_property='value')]
)
def update_output_Province(intervals, inputValueProvinces, inputValueNode):
    triggered_id=ctx.triggered_id
    #Since I have two inputs, I used a ctx to know which input was selected
    #This function basically gets the filter value that the user selects from  
    #provinces or my-input ids and then updates my 9 graphs. Additionally, I also check whether to update
   
    global df #kept this as a global variable since I reference it in my function
    global firstDate
    triggered_id=ctx.triggered_id
    if triggered_id=='provinces': #If the user selects provinces
        if inputValueProvinces!='All Provinces':
            filtered_df=df[df['Province'] == inputValueProvinces]
        else: 
            filtered_df=df
        
    elif triggered_id=='my-input': #if user selects dropdown2
        if inputValueNode!='All Nodes':
            filtered_df=df[df['NodeId'] == inputValueNode]
        else: 
            filtered_df=df
    
    elif triggered_id=='interval-component': #if a refresh is requested        
        #update df
        print(intervals)
        
        firstdate=datetime.strptime(firstDate, "%Y-%m-%d")
        
        firstDate+=timedelta(days=1) #add 1 day each time a refresh is called to update the data
        firstDate=str(firstDate.date()) #convert back to string
        if (str(firstDate).date())=='2022-09-07':
            firstDate='2022-08-31' #go back to Aug 31 if name goes beyond a threshold 
    
    
        print(firstDate) #for error debugging
        df=collectData(firstDate)
        filtered_df=df
        print(filtered_df) #this only gets printed SOME times, when it should be updating the graph each time
    else: #I kept this instance because sometimes (idk why) the trigger_id != any of my inputs 
        df=collectData("2022-08-31")
        filtered_df=df   

    fig=px.scatter(filtered_df, x='Time', y='Acc_InitialERabEstabSuccRate', color='NodeId', hover_name='NodeId')

    fig2=px.scatter(filtered_df, x='Time', y='Int_DlThroughput_kbps', color='NodeId', hover_name='NodeId')
    
    return fig, fig2, fig3, .....

The problem is that NO update occurs and I don’t receive any errors, when I try printing the interval count in my callback function, the value randomly moves instead of moving from 1,2,3,4. Additionally, it seems that the print statement for printing the df only works sometimes. I’m so confused why this isn’t working. Any help would be much appreciated.