Issue with Sharing Data Between Callbacks

Dash Python
1

I followed the documentation of sharing data but could not get the chart to update based on dropdown.

https://dash.plotly.com/sharing-data-between-callbacks

I integrated Dash with Django with django-plotly-dash and a separate mysql read-only data source which is working. It has multiple pages of charts. I ran into memory issue as each page loads a whole table from the database at the beginning of the dash app. I tried to reduce memory by selecting only the columns needed from the database and also change the dtypes after loading into pandas. But this is not enough especially as more records are added to the database.

So I tried to restructure the app to select a subset of the database table only.

The structure is like this.
Callback 1: a hidden input value takes default_site from user setting and output to site-dropdown value. (Site-dropdown options are populated by a static list. )
Callback2: use site-dropdown value to populate compare dropdown options. Basically site list minus site-dropdown value.
Callback3: use site-dropdown value, compare value, date picker as inputs to retrieve data from database and output to a hidden div. Output genderLabels to gender-dropdown options.
Callback 4: use the hidden div data and gender-dropdown value to populate category-dropdown options.
Callback 5: use the hidden div data as well as gender, category dropdowns (initial value “ALL” so not filtered) to generate a graph.

The app generates an initial graph but if I select another site from site-dropdown or select a site from compare, the graph does not change. I can see the gender, category dropdown options change accordingly and data is passed to hidden div. I don’t know why the graph does not change .

I tried another structure.
Callback 1, 2: the same as the above.
Callback3: use site-dropdown value, compare, date picker as inputs to retrieve data from database, use gender, category dropdowns (initial value “ALL” so not filtered) to filter the data and then generate a graph.
Callback 4, 5: access database again to retrieve data to populate gender and category dropdown options.
In this way everything works as expected but it is quite slow. But I don’t know if it is a bad idea to make multiple calls to DB on one page. Every time filter changes it takes about 5 seconds to show the graph.

Is there any suggestion on how to structure the app to make it memory efficient?
Any idea to make it faster in both structures?
Any idea why the graph is not updating in the 1st structure? Here is the code in the 1st structure.

app.layout = html.Div([

    dcc.Input(id='default-input', type='hidden'),

    html.Label('Site'),
    dcc.Dropdown(
        id='site-dropdown',
        options=[{'label': site, 'value': site} for site in sites],
                ),

    html.Label('Compare Site'),
    dcc.Dropdown(
         id='compare',
                ),

    html.Div(id='json-data', style={'display': 'none'}),

    html.Label('Gender'),
    dcc.Dropdown(
        id='gender-dropdown',
         value='ALL',
                ),

    html.Label('Category'),
    dcc.Dropdown(
          id='category-dropdown',
          value='ALL',
                ),

    html.Label('Select Dates'),
    dcc.DatePickerRange(
        id='my-date-picker-range',
        initial_visible_month=start,
        start_date=start,
        end_date=end,
        display_format="D/M/YYYY"),

    dcc.Graph(id='graph1'),
])

# callback3
@app.callback(
    [Output('json-data', 'children'),
      Output('gender-dropdown', 'options')],
    [Input('my-date-picker-range', 'start_date'),
     Input('my-date-picker-range', 'end_date'),
     Input('site-dropdown', 'value'),
     Input('compare', 'value')])
def get_data(start_date, end_date, site_id, compare):
    if compare:
         #select data between start_date and end_date and with site_id and compare
    else:
         #select data between start_date and end_date and with site_id
    data = df.to_json(date_format='iso', orient='split', index=False)
    genderLabels = [{'label': i, 'value': i} for i in df['gender'].unique()]
    return data, genderLabels

#callback 4
@app.callback(
    Output('category-dropdown', 'options'),
    [Input('json-data', 'children'),
     Input('gender-dropdown', 'value')])
def set_category_options(data, selected_gender):
    df= pd.read_json(data, orient='split') 
    if selected_gender == "ALL":
        categories =df['category'].unique()
    else:
        categories = df[df['gender'] == selected_gender]['category'].unique()
    categoryLabels = [{'label': i, 'value': i} for i in categories]
    categoryLabels.append({'label': 'ALL', 'value': 'ALL'})
    return categoryLabels

#callback 5
@app.callback(
    Output('graph1', 'figure'),
    [Input('json-data', 'children'),
     Input('site-dropdown', 'value'),
     Input('compare', 'value'),
     Input('gender-dropdown', 'value'),
     Input('category-dropdown', 'value')]
)
def update_figure(data, site_id, compare, gender, category):
    if compare:
         df = df[df['site_id']==compare]
         ....
    else:
        df = df[df['site_id'] == site_id]

    #filter by gender and category depending on values...
    #generate graph
    return figure