A nonexistent object was used in an `Output` of a Dash callback - correct use of multi-Page App Validation

Dash Python
1

Hi Plotly community,

I’ve been trying to migrate my one page tab based dashboard to a multi-page app with tabs/sub-tabs in pages. However running into a rather stubborn problem:

A nonexistent object was used in an `Output` of a Dash callback. The id of this object is `plotly_figure_1` and the property is `figure`. The string ids in the current layout are: [table_1, table_2, message_string....etc]

Here is my stack post a few days ago: https://stackoverflow.com/questions/77167192/dash-multi-page-app-failing-to-detect-callback-output-ids-in-different-page-layo

I’ve struggled to reproduce the error with a MWE that emulates my use case. I don’t believe it’s the way I’ve modularised my code into different folders, as the app renders and routes exactly as I expect.

The issue comes down to callback output IDs referenced on only certain pages.

My work-around renders the app unusable (referenced in my stack post as “Bad solution”). I create a universal store where all my callback output ID’s are assigned to individual dcc.Store() objects including plots. This collection of stores (x99) is wrapped in a div, which I then pass into each of the indidual page layouts. However, this is inefficient as plots are now in stores, which then render very slowly. In my original approach, this same collection of dcc.Store() objects (x10) contain only dataframes and string variables.

Here is a MWE adapted from @AnnMarieW examples for mutli-page apps, that implements validation and modularises the code into different folders. I’m using Dash v2.13.0 and Plotly v5.14.1.

Folder structure:

- app.py 
- app_scripts
    - app_pages       
       |-- __init__.py
       |-- barcharts.py
       |-- heatmaps.py
       |-- histograms.py
    - callbacks
       |-- all_callbacks.py
       |-- callbacks.py
    - navigation_bar
       |-- nav_bar.py

app.py

from dash import Dash, dcc, html, Input, Output, callback
import dash_bootstrap_components as dbc

from app_scripts.pages import barcharts, heatmaps, histograms
from app_scripts.callbacks.all_callbacks import all_callbacks
from app_scripts.navigation_bar.nav_bar import create_navbar

app = Dash(__name__,
           external_stylesheets=[dbc.themes.BOOTSTRAP]
           )

# Create navigation bar.
navbar = create_navbar()

# App contents.
url_bar_and_content_div = html.Div([
    navbar,
    dcc.Location(id='url', refresh=False),
    html.Div(id='page-content')
])

# Index layout.
app.layout = url_bar_and_content_div

# Validate complete layout.
app.validation_layout = html.Div([
    url_bar_and_content_div,
    barcharts.layout,
    heatmaps.layout,
    histograms.layout
])

# Index callbacks
@callback(Output('page-content', 'children'),
          Input('url', 'pathname'))
def display_page(pathname):
    if pathname == "/":
        return barcharts.layout
    elif pathname == "/heatmaps":
        return heatmaps.layout
    elif pathname == "/histograms":
        return histograms.layout

# Callbacks.
all_callbacks(app)

if __name__ == '__main__':
    app.run(debug=True)

barcharts.py

from dash import dcc, html
import plotly.express as px

df = px.data.tips()
days = df.day.unique()

layout = html.Div(
    [
        dcc.Dropdown(
            id="dropdown",
            options=[{"label": x, "value": x} for x in days],
            value=days[0],
            clearable=False,
        ),
        dcc.Graph(id="bar-chart"),
    ]
)

heatmaps.py

from dash import dcc, html
import plotly.express as px

df = px.data.medals_wide(indexed=True)

layout = html.Div(
    [
        html.P("Medals included:"),
        dcc.Checklist(
            id="heatmaps-medals",
            options=[{"label": x, "value": x} for x in df.columns],
            value=df.columns.tolist(),
        ),
        dcc.Graph(id="heatmaps-graph"),
    ]
)

histograms.py

from dash import dcc, html

layout = html.Div(
    [
        dcc.Graph(id="histograms-graph"),
        html.P("Mean:"),
        dcc.Slider(
            id="histograms-mean", min=-3, max=3, value=0, marks={-3: "-3", 3: "3"}
        ),
        html.P("Standard Deviation:"),
        dcc.Slider(id="histograms-std", min=1, max=3, value=1, marks={1: "1", 3: "3"})
        
    ]
)

all_callbacks.py

from app_scripts.callbacks.callbacks import callbacks_function

def all_callbacks(app):
    """
    Wrapper for individual callbacks

    """
    callbacks_function()

callbacks.py

from dash import Input, Output, callback
import plotly.express as px
import numpy as np

df_1 = px.data.tips()
df_2 = px.data.medals_wide(indexed=True)

def callbacks_function():  
    # Page 1 callback
    @callback(Output("bar-chart", "figure"), Input("dropdown", "value"))
    def update_bar_chart(day):
        mask = df_1["day"] == day
        fig = px.bar(df_1[mask], x="sex", y="total_bill", color="smoker", barmode="group")
        return fig
    # Page 2 callback
    @callback(Output("heatmaps-graph", "figure"), Input("heatmaps-medals", "value"))
    def filter_heatmap(cols):
        fig = px.imshow(df_2[cols])
        return fig
    # Page 3 callback
    @callback(
    Output("histograms-graph", "figure"),
    Input("histograms-mean", "value"),
    Input("histograms-std", "value"),
    )
    def display_color(mean, std):
        data = np.random.normal(mean, std, size=500)
        fig = px.histogram(data, nbins=30, range_x=[-10, 10])
        return fig

nav_bar.py

import dash_bootstrap_components as dbc

def create_navbar():
    return(
        dbc.NavbarSimple(
            children=[
                #### Barchart page.
                dbc.Button(dbc.NavLink(
                    'Barcharts',
                    href = '/',
                    class_name = 'nav-link',
                    active = 'exact'), 
                    class_name = 'btn btn-info me-md-3 justify-content-md-end'),
         
                #### Heatmap page.
                dbc.Button(dbc.NavLink( 
                    'Heatmaps',
                    href = '/heatmaps',
                    class_name = 'nav-link',
                    active = 'exact'), 
                    class_name = 'btn btn-info me-md-3 justify-content-md-end'),
               
                #### Histogram page.
                dbc.Button(dbc.NavLink(
                    'histograms',
                    href = '/histograms',
                    class_name = 'nav-link',
                    active = 'exact'), 
                    class_name = 'btn btn-info me-md-3 justify-content-md-end'),
                
            ],
            brand = 'Data analysis app',
            color = '#939597',
            dark = True,
            fluid = True) 
        )

This structure of MWE is effectively the same as my app, except that I have a more elaborate layout scheme for each page. I have a data upload page keyed to a dash-uploader button and callback. Upon upload of zipped files, the callback chain kicks in, with plots eventually rendered on the visualisation page in sub-tabs. The dataframes in code above, are place-holders for what are uploaded files and store objects, that in turn are inputs for my actual callbacks. My callbacks follow the same structure as in MWE, some outputs are referenced in data upload page but not in visualisation page and vice versa.

Here is my current app.py that uses routing and validation in extactly the same way as MWE:

# =============================================================================
#### Import packages into environment.

import dash_uploader as du
import dash_bootstrap_components as dbc
import dash 
from dash import html, Input, Output, callback, dcc

# =============================================================================
#### Import functions.

from app_scripts.navigation_bar.nav_bar import create_navbar
from app_scripts.callbacks.all_callbacks import all_callbacks
from app_scripts.app_data_store.software_1_data_store import data_store_1
from app_scripts.app_pages import home, data_upload, visualisation, about

# =============================================================================
#### Initialise the application dashboard.

app = dash.Dash(__name__, 
                suppress_callback_exceptions = True,
                prevent_initial_callbacks = True,
                external_stylesheets = dbc.themes.BOOTSTRAP)

# --------------------------------------------------------------------------- #
#### Configure uploader.

du.configure_upload(app, 
                    r'C:\tmp\data_uploads', 
                    use_upload_id = True)

# --------------------------------------------------------------------------- #
#### Generate navigation bar.

nav_bar = create_navbar()

# --------------------------------------------------------------------------- #
#### Generate layout.

app_contents = html.Div(
    style = {'background': '#f0f0f0'},
    children = [
        data_store_1(),
        nav_bar, 
        dcc.Location(id='url', refresh=False),
        html.Div(id='page-content')
        ]
    )

#### Index layout.
app.layout = app_contents

#### Validate complete layout.
app.validation_layout =  html.Div([
    app_contents,
    home.home_page_layout,
    data_upload.upload_page_layout,
    visualisation.visualisation_page_layout,
    about.about_page_layout
    ])

#### Index router callbacks.
@callback(Output('page-content', 'children'),
          Input('url', 'pathname'))
def display_page(pathname):
    if pathname == "/":
        return home.home_page_layout
    elif pathname == "/data_upload":
        return data_upload.upload_page_layout,
    elif pathname == "/visualisation":
        return visualisation.visualisation_page_layout,
    elif pathname == "/about":
        return about.about_page_layout
    
# --------------------------------------------------------------------------- #
#### Specify callbacks.

all_callbacks(app)

# --------------------------------------------------------------------------- #
if __name__ == '__main__':
    app.run_server(debug = True, 
                   dev_tools_hot_reload = False)
    
# --------------------------------------------------------------------------- #

Question:

1 - Why doesn’t MWE structure generate this error: A nonexistent object was used in an Output of a Dash callback, when the output ID’s in callbacks are only referenced in specific page layouts?

If I comment out the app.validation_layout lines, then there are some errors, but referencing callback inputs. Passing suppress_callback_exceptions=True to the Dash instance, suppresses these errors.

However, suppressing exceptions doesn’t work in my actual app.

2 - Does the app.validation_layout actually help with output errors?

Thanks for taking the time and apologies for the long post. I’ve spent a couple of weeks working on this and I’m loathe to give up. Hopefully one of you can save me from this rabbit hole haha!

2

Hi @AlexM

Is there a reason why you aren’t using Pages? It makes creating multi-page apps much easier. Looks like you might already know about this project, but you’ll find lots of good examples here:

3

Hi @AnnMarieW,

Thanks for engaging with my issue. My first iteration used the newest approach as described in your examples and in: https://dash.plotly.com/urls#example:-simple-multi-page-app-with-pages

It was the simplest to code and my preferred option. However, I still get exactly the same result i.e the same list of missing outputs when I’m on a specific page. I tried to implement validation with this approach too, but again same issue.

I tried splitting the initiating part of app.py ie app = Dash(__name__, use_pages = True,....) into it’s own file, and then called the rest of the app layout including if __name__ == '__main__': app.run(debug=True) from index.py. This gave the same result.

I also tried to pass the prevent_initial_call = True attribute to all callbacks. This prevented the errors when I’m navigating around pages, but the errors appear when the callbacks with outputs referenced on specific pages, are initiated. I know this because I coded each callback (x21) to have a loading element animation section on the upload page. As soon as one finishes, a “complete” message is passed to the app to show that various data upload, data processing and plot generation callbacks have completed. In my one page with tabs approach, as each plot generation callback completes, I can view plots in another tab, whilst the others complete. That was the functionality I was hoping to port to multi-pages.

The reason I used the older approach for my MWE, is that it was how the example for validation_layout() was structured. I thought that validating a layout exactly as specified was the key to this.

Frustrating thing is that it works, even if badly, with this universal data store approach. I just can’t get my head round how to convince the app that the outputs are there, without passing everything to a store object.

Perhaps it’s the way I validate the layout? I expected that would declare a complete app layout, as described in Dash docs.

BTW i just realised I missed the 3rd page (histogram) in my post. I will edit that in after this.

I also didn’t specify that I’m using the latest Dash and plotly versions.

Thanks for your time and appreciate any help.

Alex

4

Came across this stack post: https://stackoverflow.com/questions/69738178/dash-app-dynamically-create-a-layout-for-multi-page-app-validation-error-a-n

The OP’s last post talks about “declaring the validation layout upfront” and places it in his callbacks script before the callbacks. However, I’m not sure how you would declare app.validation_layout = html.Div([some_layouts]) when app is in app.py.

5

Hi @AlexM

Could you post a minimal example in GitHub? Then it would be easier to see what’s causing the error.

One of the cool features of Pages, is that it automatically does the validation_layout step for you. No need to do this step, unless you are not using Pages. Note that in large apps the validation_layout function can be slow, so you can skip it by setting suppress_callback_exception=True

1 Like
6

Hi @AnnMarieW,

ah cool. Makes total sense for that to be under the hood for pages feature. I will try to come up with a MWE that brings it closer to my actual implementation. I will post on GitHub and go from there.

Thanks

1 Like
7

Hey @AlexM

I just noticed that you mentioned you were converting a single page app that used Tabs into a multi-page app. How easy this will be may depend on how you do the navigation.

If you update the tab layout content with a callback (similar to a multi-page app) then it should be easy. However, if all the tab content is in the tab Children then it can be trickier. In this case, all the content exists in the DOM and the tab component can handle the navigation by hiding or displaying content - no callback required. This makes it possible to have, for example, a Dropdown in one tab update a Graph in another tab. This will not work when you update the layout in a callback - because when you are on the page with the Graph, the Dropdown no longer exists – so you will see the error of “A nonexistent object…”

1 Like
8

Hi @AnnMarieW and community,

I got something that partially works. My typical callback is like this:

@callback([Output('plotting_complete_message', 'children'),
           Output('plot_1', 'figure'),
           Output('plot_2', 'figure')],
          [Input('dataframe_1', 'data')])
  
def plot_generator_1(dataframe_1):
       # Check if input does not contain data and prevent app update.
       if dataframe_1 is None:
            raise PreventUpdate

       # Check if input contains data and trigger callback.
       elif dataframe_1 is not None:
            some code that returns plot_1 and plot_2
 
            plotting_complete_message = html.H3("Complete")

            return(plotting_complete_message, plot_1, plot_2)
       else:
            return(no_update, no_update, no_update)

I realised something after reading this post:https://community.plotly.com/t/a-nonexistent-object-was-used-in-an-output-of-a-dash-callback/60897/18

The callbacks linked to output errors, were all like in my toy example above. The “plotting_complete_message” was referenced in my data upload page layout, passing a “Complete” message to the page, signalling that plot could be viewed in the visualisation page. The plots are figures passed to and referenced on the visualisation page layout. In other words 1 callback had outputs referenced in 2 page layouts.

I therefore split my callback into a plot generation callback and a “listener” callback:

Plot generation callback

@callback([Output('plot_generator_1_status', 'data'),
           Output('plot_1', 'figure'),
           Output('plot_2', 'figure')],
          [Input('dataframe_1', 'data')])
  
def plot_generator_1(dataframe_1):
       # Check if input does not contain data and prevent app update.
       if dataframe_1 is None:
            raise PreventUpdate

       # Check if input contains data and trigger callback.
       elif dataframe_1 is not None:
            some code that returns plot_1 and plot_2
 
            plot_generator_1_status = "complete"

            return(plot_generator_1_status, plot_1, plot_2)
       else:
            return(no_update, no_update, no_update)

Listener callback

@callback([Output('plot_complete_message', 'children')],
          [Input('plot_generator_1_status', 'data')])
    def plot_generator_1_listener(plot_generator_1_status):
        # First condition if callback not completed.
        if plot_generator_1_status is None:
            # Exception raised in order to suppress error.
            raise PreventUpdate
        
        # If callback completed, generate message.
        elif plot_generator_1_status == 'complete':
            
            plot_complete_message = html.H3('Complete')

            return(plot_complete_message)
        
        else:
            return(no_update)

I now update my global store with dcc.Store(id = plot_generator_1_status). Rinse and repeat for all callbacks with probematic dual outputs.

App now behaves as follows:

Clicking betwen pages doesn’t raise errors.

Data uploads as normal with messages regarding initial pre-processing of data displayed on upload page. However, the “plot_complete_messages” for plots don’t display in the upload page and update errors return.

What works:

If I now navigate to the “visualisation” page, the app immediately starts updating and eventually displays my plots in each of the sub-tabs. It’s about as quick as my 1 page with tabs app. It seems clicking on the “visualisation” page kickstarts my plotting callbacks. In my one page with tabs approach, the plots will render in the “visualisation” tab whilst I’m still in the upload tab, monitoring status messages. This will not work, if I remove the callback splitting method.

Additionally, the output ID errors in the “visualisation” page now don’t appear whilst plots are rendered, so half the battle won! It’s still confusing as those ID’s are still not referenced in the “visualisation” page layout.

What doesn’t work:

1 - My plot completion messages don’t update upload page. This is still due to callback output ID’s for plots only being referenced in the “visualisation” page. I reverted to new approach for pages that already has validation baked in, as @AnnMarieW mentioned in previous post, but same behaviour. Did shorten my code though!

2 - Clicking to other pages and back re-starts the callback chain. I don’t know how to suppress this refresh behaviour. Passing
storage_type='session' to dcc.Store doesn’t work, because I have no plots in stores, only tables and string variables.

3 - If I click on another browser tab while plots are updating, then go back to app whilst it’s updating plots, I sometimes get get Callback failed: the server did not respond. errors.

There are additional layers of complexity here that are beyond my skill level with Dash. Unfortunately, trying to work up a MWE that is representative of my functionality and reproduces this behaviour has defeated me. I’m going to park this for now and revert to my old approach.

If anyone has a generic template MWE of a multi-page app with data upload and visualisation pages sending data to each other, I would love to see it.

Again, apologies for long post (I’ve been down the rabbit hole for weeks and need to tell someone haha). Perhaps this will be useful for people with similar issues.

@AnnMarieW thanks for your help!

9

Hi @AnnMarieW,

just saw your message after typing my long post.

Thanks for engaging with me on this!

I need to think about your suggestion and decide if I can handle another round with my multi-page app haha.

I’m still confused however. My tabs are all pre-defined in functions, which include elements like this:

dbc.Col(html.Div(dcc.Graph(id = 'plot_1')),

I then wrap these tabs in a div that renders 1 page which Dash registers using the new pages features. The callbacks generate plots that update the tab elements above and therefore the page layout. I’m not entirely sure if this is the case you refer to, that doesn’t work. My brain is mush right now.

I’m not a professional developer, and really just learnt through your teams amazing documentation and examples.

I feel like I’m within touching distance.

Alex

10

Hi All,

I was loathe to give up, so I keep giving it one last stab. I did find a simple, though not entirely elegant, way to achieve what I wanted. It gets me ~90% of the way there.

I created hidden divs for the callback output ID’s that don’t appear in initial layouts.

Where I needed ID’s to be in my visualisation page layout, I passed this into that layout:

html.Div(
           children = [
               
               # ----------------------------------------------------------- #
               
               html.Div(style = {'display':'none'},
                        id = 'plot_complete_1_message'),
               
               # ----------------------------------------------------------- #
               
               html.Div(style = {'display':'none'},
                        id = 'plot_complete_2_message')
               
               # ----------------------------------------------------------- #
       
               ]
            )  
         )

Where I needed ID’s to be in my upload page layout, I passed this into that layout:

html.Div(
            children = [
                
                # ----------------------------------------------------------- #
                
                html.Div(style = {'display':'none'},
                         children = [dcc.Graph(id = 'plot_1')]),
                
                # ----------------------------------------------------------- #
                
                html.Div(style = {'display':'none'},
                         children = [dcc.Graph(id = 'plot_2')]),
                
                # ----------------------------------------------------------- #
        
                ]
             )  
          )

Now all missing ID’s are in the initial app layout. All loading spinners and messages display in upload page. I also got rid of the “listener” callbacks as they were now unnecessary.

However, when I navigate onto visualisation page, callback chain re-starts. The plots do render as quickly as in my 1 page with tabs app, but doubles up on time it takes to do a given analysis.

Clicking between pages re-starts callbacks for that page and so on. I was able to silence some of this with prevent_initial_call = True in my callbacks, but not entirely.

I think now that, I would have to re-factor my whole layout and callback structure to get this to work completley smoothly. I had hoped to plug and play my tabs layouts straight into pages, but obviously it wasn’t that straight forward.

I still learnt alot and gave me ideas on a better structure for my original dashboard.

Hope this helps someone down the line!

Alex

11

Hi @AlexM

Thanks for sharing your work-around!

Here’s another options you might find helpful. @Emil recently created a utility in dash-extensions that controls the visibility of the page based on the URL If you give it a try, it would be great if you could give feedback in this thread. I think this could be a nice addition to Dash as this is a pretty common use-case in multi-page apps.

12

Hi @AnnMarieW,

I looked at @Emil dash extension for pages post.

Hmmm…if I can persist my updated layouts without the refresh when clicking between pages, that’s 95% there.

What I’m still trying to understand, is why the plotting callbacks initiate again when I click into the visualisation page. They are already triggered on the upload page, and when all the complete messages appear, then I would like to see visualisation page already populated.

Other than this re-trigger, I don’t think there is much over-head in the callbacks leading to plots rendering. My biggest over-head, even with original dashboard, is converting plotly figures to pngs which are then served to the app (some of my plots consist of 100K+ points). Moved from Kaleido to Orca, as it was abit faster for this purpose.

Right…into the breach once more! I will find some time over the next week and post back.

Alex

13

Hi @AlexM

You might be interested in this section of the docs: When are callbacks executed?

14

I am using pages with your example. I made very nice app for my engineering task. could you please help me figure out what is the problem with my app?