I have a Multi page dash app which just stops running a few callbacks that trigger based on a dcc.store object

Hi Everyone,

I hope you are doing great, currently I am working on creating a dash application which uses a SARIMAX model to predict the hourly incoming volume @ a warehouse. There are bunch of different tabs in my application as per the use case, but all in all everything used to work fine since I started working on the app a few months ago.

In a couple of my tabs I have 3 dash tables each of which is dependent on the previous one. So the first dash table updates based on a dcc.interval object every 10 seconds & the data is then updated. The next callback takes the data of the first table as an input & updates the second table & so on. The app works just fine normally but in the last few recent days, suddenly it would just start showing empty tables from my second dash table & every call back after that stop working. After a few minutes, it would be resolved & then everything would start working normally.

I am not really sure what could be the reason for this, but I need to find a fix soon. Another interesting thing about the problem is that this app is stored on an ec2, but when this problem occurs both my dev desktop link & the local host start having the same issue & when it is working it will become normal both locally & on the dev desktop. As there are no callback errors or anything shown on debug console, it’s very difficult to figure out what’s causing this to happen? Any help here is much appreciated.

Hi @Anirudh217 ,

Welcome.

From the statement ‘when this problem occurs both my dev desktop link & the local host start having the same issue’ I assume that it might be a problem fetching the data.

Where is the data located and how do you fetch it? Is it over the internet / through an API / vpn / …
How long does the interruption last? (a few minutes if I understood?)
Do you have any error handling in your code
Are you catching connection/handling. errors / empty data on the first table? It might be that in the case of no data an empy valid data structure might be returned so that it doesn’t crash. so if you don’t log or have no specific logic it would go unnoticed. except no output

It would be nice if you could actually post some code.

So I can post the callback code here but not the entire app structure if that makes sense. Also, it is not like the app starts showing the tables after a few minutes sometime it just doesn’t while other times everything works normally. Also, the data is only a static csv file which is refreshed once daily but the app only reads a static csv file. I can post the three callbacks where the issue occurs if that would be helpful.

The following block of code defines the layout of one of my tabs, there is a radio button in my index file which switches between day & night shift. So I have 4 files similar to this, & all of them experience the same issue when this happens. Otherwise everything works fine.

staffing_layout_2da = dbc.Container([

    dbc.Row(
        [
            dbc.Col(
                modelHeader,
                width={'size': 3, 'offset': 0}),
            dbc.Col(
                html.Div(html.P('STEP 2--> View the current mapping', style={"color": "blue", 'fontSize': 13})),
                width={'offset': 1, 'size': 4}),
            dbc.Col(
                headcountHeader,
                width={'offset': 1}),
        ]),
    html.Br()
    ,
    dbc.Row(
        [
            dbc.Col(modelDropdown, width={'size': 3, 'offset': 0}),
            dbc.Col(dbc.Card(card_content, color="light", inverse=True, className='text-center'),
                    width={'size': 3, 'offset': 1}),
            dbc.Col(html.Div(id="output-data-upload_2da", style={"color": "red", 'fontSize': 12},
                             className='text-center mb-4'),
                    width={'size': 2, 'offset': 0}),
            dbc.Col(dbc.Input(
                placeholder="Headcount at ULD loading", size="sm", className="mb-3", value=100, id='headcount_2da'
            ), className='text-center success', width={'size': 3, 'offset': 0})
        ]),
    dbc.Row(
        [
            dbc.Col(download, className='text-center mb-4', width={'size': 1, "offset": 5}),
            # dbc.Col(upload, className='text-center mb-4', width={'size': 2, "offset": 0}),
        ],
        className="mb-4",
    ),
    # dbc.Row(dbc.Col(html.Div(id='map_1_2da', children=[]), width={'offset': 5})),
    # dbc.Row(dbc.Col(html.Div(id='map_2_2da', children=[]), width={'offset': 5})),
    # dbc.Row([dbc.Col(html.H2('Current Mapping   ---->', style={"color": "blue"}), width={'size': 4, 'offset': 0}),
    #          dbc.Col(html.Div(id='map_3_2da', children=[]), width={'offset': 1})]),
    # dbc.Row(dbc.Col(html.Div(id='map_4_2da', children=[]), width={'offset': 5})),
    # dbc.Row(dbc.Col(html.Div(id='map_5_2da', children=[]), width={'offset': 5})),
    # dbc.Row(dbc.Col(html.Div(id='map_6_2da', children=[]), width={'offset': 5})),
    dbc.Row(
        [dbc.Col(html.Div(id='line-container_2da'), width=6)
         ]),
    html.Br(),
    dbc.Row(
        [
            dbc.Col(html.H5(id='staffingTitle_2da', children=staffingRate['title'], style={"color": "orange"},
                            className='text-center bold mb-4'),
                    width={'size': 6, 'offset': 0}),
            dbc.Col(chartDropdown
                    , className='text-center', width={'size': 2, 'offset': 0}),
        ]
    ),
    dbc.Row(
        [dbc.Col(html.Div(id='legend_2da', children=[], style={'float': 'left'}), width={'offset': 3}),
         ]),
    dbc.Row(
        [dbc.Col(staffingRate['object'], width=6),
         dbc.Col(html.Div(id='line-container_2da'), width=6)
         ]),
    dbc.Row(
        [dbc.Col(html.H5(id='stackingTitle_2da', children=topStacking['title'], style={"color": "orange"},
                         className='text-center bold mb-4'),
                 width=6)
         ]),
    dbc.Row(
        [dbc.Col(topStacking['object'], width=12)]),
    html.Br(),
    dbc.Row(
        [dbc.Col(html.H5(id='flightTitle_2da', children=divertVol['title'], style={"color": "orange"},
                         className='text-center bold mb-4'),
                 width=6),
         dbc.Col(html.H5(id='fingerTitle_2da', children=fingerVol['title'], style={"color": "orange"},
                         className='text-center bold mb-4'),
                 width=6)
         ]),

    dbc.Row(
        [dbc.Col(divertVol['object'], width=6),
         dbc.Col(fingerVol['object'], width=6),
         ]),
    dcc.Store(id='store-flight-data_2da', data=[], storage_type='memory'),
    dcc.Store(id='store-finger-data_2da', data=[], storage_type='memory'),
    dcc.Store(id='store-rates_2da', data=[], storage_type='memory'),
    dcc.Interval(
        id='my_interval',
        disabled=False,  # if True, the counter will no longer update
        interval=1 * 6000,  # increment the counter n_intervals every interval milliseconds
        n_intervals=0,  # number of times the interval has passed
        max_intervals=-1,  # number of times the interval will be fired.
        # if -1, then the interval has no limit (the default)
        # and if 0 then the interval stops running.
    ),
    dcc.Interval(
        id='my_interval_1',
        disabled=False,  # if True, the counter will no longer update
        interval=20 * 6000,  # increment the counter n_intervals every interval milliseconds
        n_intervals=0,  # number of times the interval has passed
        max_intervals=-1,  # number of times the interval will be fired.
        # if -1, then the interval has no limit (the default)
        # and if 0 then the interval stops running.
    )
], fluid=True)

# *************************************************************
""" CALLBACKS """


# *************************************************************

# Callback to store the flight refresh the flight & rates data periodically so this will ensure the data is refreshed
# properly & we are able to detect the daily backend data updates

@app.callback(
    [Output('datatable-interactivity-1_2da', 'data'),
     Output('datatable-interactivity-1_2da', 'columns'),
     Output('flightTitle_2da', 'children'),
     Output('fingerTitle_2da', 'children'),
     Output('staffingTitle_2da', 'children')],
    [Input('my_interval', 'n_intervals'),
     Input('modelDropdown_2da', 'value')]
)
def flightData(num, modelSelection):
    print(num)
    print('***************')
    if num == 0:
        if modelSelection == 'sorttech cpt view + wwde ML forecast (recommended)':
            flightForecast, currentDate = flightProcessor()
        else:
            flightForecast, currentDate = flightProcessor(cpt=False)

        # need to return the cols & data for the first dash table
        col = [{"name": i, "id": i, "deletable": False, "selectable": False, "hideable": False}
               for i in flightForecast.columns
               if i != 'id'
               ]
        var1 = f'Expected Divert Volume per flight - {currentDate}'
        var2 = f'Expected Divert Volume per finger - {currentDate}'
        var3 = f'Hourly Staffing recommended per finger - {currentDate}'

        return flightForecast.to_dict('records'), col, var1, var2, var3
    else:
        if modelSelection == 'sorttech cpt view + wwde ML forecast (recommended)':
            flightForecast, currentDate = flightProcessor()
        else:
            flightForecast, currentDate = flightProcessor(cpt=False)

        col = [{"name": i, "id": i, "deletable": False, "selectable": False, "hideable": False}
               for i in flightForecast.columns
               if i != 'id'
               ]
        var1 = f'Expected Divert Volume per flight - {currentDate}'
        var2 = f'Expected Divert Volume per finger - {currentDate}'
        var3 = f'Hourly Staffing recommended per finger - {currentDate}'

        return flightForecast.to_dict('records'), col, var1, var2, var3


# Visualizing the expected divert Volume across the shift

# Consuming the new flight finger mapping uplaoded by the user
@app.callback(
    [Output('output-data-upload_2da', 'children'),
     Output('datatable-interactivity-2_2da', 'data'),
     Output('datatable-interactivity-2_2da', 'columns')],
    [Input('datatable-interactivity-1_2da', 'data'),
     Input('my_interval_1', 'n_intervals')]
)
def store_data(flightData, num):
    # hypothetical enormous dataset with millions of rows
    if num == 0:
        flight = pd.DataFrame(flightData)
        print(flight)
        print('**********')
        fingerForecast, fileMsg = fingerProcessor(flight)

        col = [{"name": i, "id": i, "deletable": False, "selectable": False, "hideable": False}
               for i in fingerForecast.columns
               if i != 'id'
               ]

        print(fileMsg)

        return fileMsg, fingerForecast.to_dict('records'), col
    else:
        flight = pd.DataFrame(flightData)
        print(flight)
        print('**********')
        fingerForecast, fileMsg = fingerProcessor(flight)

        col = [{"name": i, "id": i, "deletable": False, "selectable": False, "hideable": False}
               for i in fingerForecast.columns
               if i != 'id'
               ]

        print(fileMsg)

        return fileMsg, fingerForecast.to_dict('records'), col


# Giving an option to the user for downlaoding the current mapping
@app.callback(
    Output("download-mapping_2da", "data"),
    Input("btn-download-txt_2da", "n_clicks"),
    prevent_initial_call=True,
)
def downloadFunc(n_clicks):
    return dcc.send_file('Finger_flight_mapping_2DA.xlsx')


# https://stackoverflow.com/questions/69916523/dash-callback-taking-multiple-inputs-where-some-may-or-may-not-exist-depending
@app.callback(
    [Output(component_id='datatable-interactivity-3_2da', component_property='data'),
     Output(component_id='datatable-interactivity-3_2da', component_property='columns'),
     Output(component_id='datatable-interactivity-3_2da', component_property='style_data_conditional'),
     Output(component_id='legend_2da', component_property='children')],
    [Input(component_id='headcount_2da', component_property='value'),
     Input(component_id='datatable-interactivity-2_2da', component_property='data')
     ]
)
def updateStaffing(headcount, data):
    # https://stackoverflow.com/questions/72229774/schema-length-validation-error-on-python-dash-datatable
    data = pd.DataFrame(data)
    """
           Diving each element of the dataframe by the total volume convert it to percentage
           & multiply by the headcount & round it to the next integer
    """

    if headcount:
        data_temp = data.drop(['id'], axis=1)
        data_temp.set_index(['Finger'], inplace=True)
        headcount = int(headcount)
        data_temp = data_temp.div(data_temp.sum(axis=0), axis=1)
        data_temp = data_temp.apply(lambda x: np.round(x * headcount, 0))
        (styles, legend) = discrete_background_color_bins(data_temp)
        data_temp = data_temp.reset_index()
        col = [
            {"name": i, "id": i, "deletable": False, "selectable": False, "hideable": False}
            for i in data.columns
            if i != 'id'
        ]
        print(data_temp)
        # data_temp.loc['Grand Total'] = data_temp.sum(numeric_only=True, axis=0) - todo -> this causes the staffing
        #  line chart to break fix it
        return data_temp.to_dict('rows'), col, styles, legend
    else:
        raise PreventUpdate


# Create bar chart
@app.callback(
    [Output(component_id='line-container_2da', component_property='children')],
    [Input(component_id='datatable-interactivity-1_2da', component_property='selected_rows'),
     Input(component_id='datatable-interactivity-2_2da', component_property='selected_rows'),
     Input(component_id='datatable-interactivity-3_2da', component_property='selected_rows'),
     Input(component_id='datatable-interactivity-1_2da', component_property='data'),
     Input(component_id='datatable-interactivity-2_2da', component_property='data'),
     Input(component_id='datatable-interactivity-3_2da', component_property='data'),
     Input(component_id='chartDropdown_2da', component_property='value'),
     Input('flightTitle_2da', 'children'),
     Input('fingerTitle_2da', 'children'),
     Input('staffingTitle_2da', 'children')
     ]
)
def update_line(selected_rows_1, selected_rows_2, selected_rows_3,
                flightData, fingerData, staffingData, chartParam, var1, var2, var3):
    print('***************************************************************************')
    # print('Data across all pages pre or post filtering: {}'.format(all_rows_data))
    # print("Indices of selected rows regardless of filtering results: {}".format(selected_rows_1))
    # print("Indices of selected rows regardless of filtering results: {}".format(selected_rows_2))
    if chartParam == 'Flight':
        dff = pd.DataFrame(flightData)

        if selected_rows_1:
            dff = dff.iloc[selected_rows_1]
        # else:
        #     dff = dff.iloc[order_of_rows_indices]
        # # We need to unpivot the data frame , the line below will make the plotting simpler
        dff = dff.reset_index()
        # print(dff['Flight'])
        dff = dff.drop(['id'], axis=1)  # to remove the unneccesary string fields
        dff = dff.drop(['index'], axis=1)  # to remove the unneccesary string fields
        dff_unpivoted = dff.melt(id_vars=['Flight'], var_name='time', value_name='Package Count')
        fig = px.line(dff_unpivoted, x='time', y='Package Count', color='Flight')
        fig.update_layout(
            yaxis_title='Package Count',
            title="<b>" + var1 + "</b>"
        )

        return [
            dcc.Graph(id='line-chart_2da',
                      figure=fig)
        ]
    elif chartParam == 'Staffing':
        dff = pd.DataFrame(staffingData)

        if selected_rows_3:
            dff = dff.iloc[selected_rows_3]
        # # We need to unpivot the data frame , the line below will make the plotting simpler
        dff = dff.reset_index()
        # print(dff['Finger'])
        dff = dff.drop(['index'], axis=1)  # to remove the unneccesary string fields
        dff_unpivoted = dff.melt(id_vars=['Finger'], var_name='time', value_name='Associate Count')
        fig = px.line(dff_unpivoted, x='time', y='Associate Count', color='Finger')
        fig.update_layout(
            yaxis_title='Associate Count',
            title="<b>" + var3 + "</b>"
        )

        return [
            dcc.Graph(id='line-chart_2da',
                      figure=fig)
        ]


    else:
        dff = pd.DataFrame(fingerData)

        if selected_rows_2:
            dff = dff.iloc[selected_rows_2]
        # print(dff)
        # else:
        #     dff = dff.iloc[order_of_rows_indices]
        # # We need to unpivot the data frame , the line below will make the plotting simpler
        dff = dff.reset_index()
        dff = dff.drop(['index'], axis=1)  # to remove the unneccesary string fields
        dff = dff.drop(['id'], axis=1)  # to remove the unneccesary string fields
        dff_unpivoted = dff.melt(id_vars=['Finger'], var_name='time', value_name='Package Count')
        fig = px.line(dff_unpivoted, x='time', y='Package Count', color='Finger')
        fig.update_layout(
            yaxis_title='Package Count',
            title="<b>" + var2 + "</b>"
        )

        return [
            dcc.Graph(id='line-chart_2da',
                      figure=fig)
        ]

So as you can see here I update the first callback through a function called flight processor which is just reading a static csv file & then all the subsequent callbacks till the graph creation are dependent on that.

@Anirudh217

It’s hard to say something useful about this TBH.

I just notice that there is no minimal error checking regarding the data. returned by any of the data functions.
As there is no visibility in these functions, nor visible error handling, I suggest to start there.

  • What do you get back (is it valid data : nothing / strange dates / strange names / … ) / how many (how many rows : 0? or abnormal many compared to usual ?
  • How long does each processing take (normal, without issues) “Millions of records …?”
    (the interval is set to 120s),
  • etc …

IMHO, the only way to figuring this out is to start measuring and logging

And usually the question : “What has changed” is the most important one, if it started happening a few days ago

1 Like

Sometimes it just shows me loading … & nothing happens. I am wondering if there’s anything why does it sometime work while the other time stops without showing any error? Can this be a throttling issue

I think I might need to get in touch with an expert to resolve the issue. Would it be possible for you to set up some time with me to go over the problem?

@Anirudh217

I don’t consider myself an expert by any measure, but thanks anyway :slight_smile:

What I proposed is just part of a common sense troubleshooting approach, Which I think is the minimum needed to get your problem resolved.

In general, this is a community support where people try to help others within some reasonable measure and with enough feedback provided.

If you require personal /dedicated support you might need to look at some commercial form of support.
Maybe @adamschroeder can point you in the right direction for such support

Hi @Anirudh217 welcome to the forums. I formated your code using the preformatted Text icon. Please consider doing that everytime you post code on this forum, it helps reading your code :slight_smile:

Are you running your app with debug=True?

hi @Anirudh217
Hopefully, we are successful in supporting you in this topic thread.

Another option is to offer an opportunity for community members looking for work. You could post your question with the #job-postings tag and let people know how to contact you to set up this opportunity. Let me know if you need help doing that.

1 Like