Queries running twice on page load

Hi all,

I have four sql queries that I need to run every times the page loads (user refreshes). I have tried multiple ways to make this work so that the queries only run once, but I haven’t been able to figure it out. I have a serve_layout function and I have placed the queries outside the serve_layout. (When they were in serve_layout they were running three times so I moved them out.)

By my logging, the queries get called even before the app launches, and then once it has launched. Any thoughts on how to fix this? I have one query that takes a while so that’s what’s driving me nuts.

Thanks!

(Logging for reference)

Hello @bpolasek,

As the app spins up, all the layouts are ran through and populated for testing.

If you dont want these to be called, then you can disable those sections inside the respective page layouts.

I’m not sure what you mean.

Also it’s only a one page app so I don’t have multiple pages.

Here is part of my code if that helps:

def serve_layout():
    layout = html.Div([
        nav,
        dcc.Store(id='store', data={'df': all_with_areas_names_df.to_dict('records')}),
        dcc.Store(id='four_tables', data={'df': cat_with_names_df.to_dict('records')}),
        dcc.Store(id='alarm-table', data={'df': all_alarm_data_w_names.to_dict('records')}),
        dcc.Store(id='limit-table', data={'df': limit_table.to_dict('records')}),
        dcc.Download(id='limit-csv'),
        dbc.Container([
            dbc.Row([
                dbc.Col([
                     ....

    return layout


current_time = dt.now()
logger.info('Querying Water Chemistry Data')
all_df = pod.get_water_chem_data()
logger.info('Done')

logger.info('Querying Name Data')
name_df = asd.get_region_engine_names()
logger.info('Done')

prev_day = current_time - timedelta(days=1)
prev_day_str = prev_day.strftime('%Y-%m-%d %H:%M:%S')
logger.info('Querying EMS Data')
osi_key_df = ems.get_osi_key(prev_day_str, get_color_logger('EmsQueryLogger', True))
logger.info('Done')

logger.info('Querying Alarm Data')
all_alarm_data = pod.get_all_alarm_data()
logger.info('Done')

dataframe maniupulation...

app.layout = serve_layout

callbacks...

Put all of those queries inside of the function, that should stop it from down it at spin up and only upon page request.

So I did have them inside the serve_layout function to begin with and it still ran it multiple (3) times.

Add a test to make sure there is an actual request:

from flask import request

if request:
    # run queries

This is what it looked like before:

def serve_layout():
    current_time = dt.now()
    prev_60_days = current_time - timedelta(days=60)
    current_time_str = current_time.strftime('%B %d, %Y - %I:%M%p')
    current_day_str = current_time.strftime('%Y-%m-%d')
    prev_60_days_str = prev_60_days.strftime('%Y-%m-%d')
    logging.info('Querying Water Chemistry Data')
    all_df = pod.get_water_chem_data()
    columns = ['Area_0', 'Area_1', 'Area_2', 'Area_3', 'Area_4']
    split = pd.DataFrame(all_df['Description'].str.split(pat=r'\|+', n=4, regex=True).to_list(), columns=columns)
    all_with_areas_df = pd.concat([all_df, split], axis=1)
    logging.info('Querying Name Data')
    name_df = asd.get_region_engine_names()
    all_with_areas_names_df = pd.merge(all_with_areas_df, name_df, how='inner', left_on='PRD_PlantAbbr',
                                       right_on='PlantAbbr')

    prev_day = current_time - timedelta(days=1)
    prev_day_str = prev_day.strftime('%Y-%m-%d %H:%M:%S')
    logging.info('Querying EMS Data')
    osi_key_df = ems.get_osi_key(prev_day_str)
    all_with_areas_names_osi_df = pd.merge(all_with_areas_names_df, osi_key_df, how='inner',
                                           left_on='EMS_Gross_MW_Point', right_on='OSI_KEY')
    cat_with_names_df = all_with_areas_names_osi_df.dropna(subset=['EquipmentType', 'PointType', 'Location']).copy()
    cat_with_names_df.reset_index(drop=True, inplace=True)
    cat_with_names_df['Cat_Type'] = np.where(cat_with_names_df['Can_Trip_Alarm'] == 1, 'Critical', 'All')
    cat_with_names_df['OOS'] = np.where(cat_with_names_df['Op_Level'] == 1, 'OOS', '')
    cat_with_names_df['Attn'] = np.where(cat_with_names_df['Op_Level'] > 1, 'Need(s) Attention', '')

    conditions = [
        (cat_with_names_df['Can_Trip_Alarm'] == 1) & (cat_with_names_df['Op_Level'] == 1),
        (cat_with_names_df['Can_Trip_Alarm'] == 1) & (cat_with_names_df['Op_Level'] > 1),
        (cat_with_names_df['Can_Trip_Alarm'] == 0) & (cat_with_names_df['Op_Level'] > 1),
        (cat_with_names_df['Can_Trip_Alarm'] == 0) & (cat_with_names_df['Op_Level'] == 1),
        cat_with_names_df['Op_Level'] == 0,
        cat_with_names_df['Op_Level'].isna()
    ]
    labels = [
        'Critical - Out of Spec.',
        'Critical - Needs Attn.',
        'All - Needs Attn.',
        'All - Out of Spec.',
        '',
        ''
    ]
    cat_with_names_df['Prefix'] = np.select(condlist=conditions, choicelist=labels)
    cat_with_names_df['Full_Description'] = cat_with_names_df['Prefix'] + ' ' + cat_with_names_df['Description']
    urgency_levels = {
        'Critical - Needs Attn.': 0,
        'Critical - Out of Spec.': 1,
        'All - Needs Attn.': 2,
        'All - Out of Spec.': 3,
        '': 4
    }
    cat_with_names_df['Prefix_Level'] = cat_with_names_df['Prefix'].map(urgency_levels)
    logging.info('Querying Alarm Data')
    all_alarm_data = pod.get_all_alarm_data()
    all_alarm_data_w_names = pd.merge(all_alarm_data, name_df, how='inner', left_on='PRD_PlantAbbr',
                                      right_on='PlantAbbr')

    all_with_areas_names_df['Short_Description'] = all_with_areas_names_df[['Area_1', 'Area_2', 'Area_3', 'Area_4']] \
        .agg('||'.join, axis=1)
    limit_table = all_with_areas_names_df[['Area_0', 'Short_Description', 'Unit', 'Norm_Min', 'Norm_Max', 'L1_Min',
                                           'L1_Max', 'L2_Min', 'L2_Max']].copy()
    limit_table.rename(columns={'Area_0': 'Plant', 'Short_Description': 'Point', 'Norm_Min': 'Norm Min',
                                'Norm_Max': 'Norm Max', 'L1_Min': 'L1 Min', 'L1_Max': 'L1 Max', 'L2_Min': 'L2 Min',
                                'L2_Max': 'L2 Max'}, inplace=True)
    limit_table.sort_values(by='Plant', inplace=True)
    limit_table.round(2)
    limit_table.fillna('N/A', inplace=True)

    layout = html.Div([
        nav,
        dcc.Store(id='store', data={'df': all_with_areas_names_df.to_dict('records')}),
        dcc.Store(id='four_tables', data={'df': cat_with_names_df.to_dict('records')}),
        dcc.Store(id='alarm-table', data={'df': all_alarm_data_w_names.to_dict('records')}),
        dcc.Store(id='limit-table', data={'df': limit_table.to_dict('records')}),
        dcc.Download(id='limit-csv'),
        dbc.Container([
            dbc.Row([
                dbc.Col([
                  ....
    return layout

app.layout = serve_layout

Okay so I did that (shown below - not sure if I implemented it correctly) and I got ‘Error Loading Layout’. It didn’t run the queries before the app launched though so theres that.

def serve_layout():
    layout = html.Div([
        nav,
        dcc.Store(id='store', data={'df': all_with_areas_names_df.to_dict('records')}),
        dcc.Store(id='four_tables', data={'df': cat_with_names_df.to_dict('records')}),
        dcc.Store(id='alarm-table', data={'df': all_alarm_data_w_names.to_dict('records')}),
        dcc.Store(id='limit-table', data={'df': limit_table.to_dict('records')}),
        dcc.Download(id='limit-csv'),
        dbc.Container([
            dbc.Row([
                dbc.Col([
                     ....

    return layout

if request: 
      current_time = dt.now()
      logger.info('Querying Water Chemistry Data')
      all_df = pod.get_water_chem_data()
      logger.info('Done')
      
      logger.info('Querying Name Data')
      name_df = asd.get_region_engine_names()
      logger.info('Done')
      
      prev_day = current_time - timedelta(days=1)
      prev_day_str = prev_day.strftime('%Y-%m-%d %H:%M:%S')
      logger.info('Querying EMS Data')
      osi_key_df = ems.get_osi_key(prev_day_str, get_color_logger('EmsQueryLogger', True))
      logger.info('Done')
      
      logger.info('Querying Alarm Data')
      all_alarm_data = pod.get_all_alarm_data()
      logger.info('Done')
      
      dataframe maniupulation...

app.layout = serve_layout

callbacks...

You need to put the if request inside the function of server_layout.

Okay so I have:

def serve_layout():
    if request:
        current_time = dt.now()
        logger.info('Querying Water Chemistry Data')
        all_df = pod.get_water_chem_data()
        logger.info('Done')

        logger.info('Querying Name Data')
        name_df = asd.get_region_engine_names()
        logger.info('Done')

        prev_day = current_time - timedelta(days=1)
        prev_day_str = prev_day.strftime('%Y-%m-%d %H:%M:%S')
        logger.info('Querying EMS Data')
        osi_key_df = ems.get_osi_key(prev_day_str, get_color_logger('EmsQueryLogger', True))
        logger.info('Done')

        logger.info('Querying Alarm Data')
        all_alarm_data = pod.get_all_alarm_data()
        logger.info('Done')

        dataframe manipulation....

    layout = html.Div([
        nav,
        dcc.Store(id='store', data={'df': all_with_areas_names_df.to_dict('records')}),
        dcc.Store(id='four_tables', data={'df': cat_with_names_df.to_dict('records')}),
        dcc.Store(id='alarm-table', data={'df': all_alarm_data_w_names.to_dict('records')}),
        dcc.Store(id='limit-table', data={'df': limit_table.to_dict('records')}),
        dcc.Download(id='limit-csv'),
        dbc.Container([
            dbc.Row([
                dbc.Col([
                    .....
    return layout

app.layout = serve_layout

And it still ran multiple times, just not before

How many tabs do you have open? XD

If you change it from debug mode, you can see the ip addresses where the requests are coming from.

Hahah only one!

And where would I see that? I turned off debug mode.

For now we just made the sql query more efficient so that’s helped, even though it calling multiple times isn’t ideal.

Is this something that needs to be queried every time?

Can you have a background process that queries it on a schedule? If so, you could store the results in a cache to be catered, just have a way to generate a timestamp if it matters that much.

Yeah so my boss wants the data to be refreshed/requeried every time the page refreshes and instead of on a time interval.

As long as this is a small scale app… and your db doesnt have tons of transactions, it should be fine then.

You’ll just suffer some performance loss.

1 Like