How does one modify a dataframe with a callback? [former: circular import errors]

Dash Python
1

Hello,

In setting up callbacks with my first dashboard in a long time, I am having trouble with “circular imports”.

Specifically, I have an app.py file that has server variables like theapp, and uses variables from index.py like lyt [short for layout]. Likewise, my index.py file is home to the lyt variable, and uses theapp to set up callbacks.

I keep getting the ImportError for circular imports, but I cannot imagine how to correctly set up my file structure so as to avoid this error. It seems like one way or another, my index file will need app files and my app file will need index files.

Does anyone have guidance on this issue?

2

Hi @plotmaster422

You can use dash.callback which isn’t dependent on the app object. More info here:

3

Thanks for the link to the documentation. I am not building an MPA, does this page still apply?

4

BTW - I am a buyer of your book, it just arrived yesterday! It’s great.

1 Like
5

Got it, it still applies - looks like one can swap @app.callback for @callback with a from dash import callback. Will give it a try (fingers crossed)

1 Like
6

@callback directly works, thank you.

However, my use-case is a bit complex, and I am still running into issues.

I am trying to filter a dataframe with a callback, and then pass that dataframe into all graphs in the dashboard.

Currently the data flow looks like this: [load_data.py → index.py with @callback to filter → feature_engineering.py for groupbys-> charts.py for charts → index.py for html]. I am hopeful this will work without the app complication.

The other minor problem for this use case is that I am having trouble importing from parent directory. It looks like it’s possible by turning it in to a package, but I’m not there yet programming-wise, and frankly it’s not a package, it’s an app. Any ideas on this?

7

A minimal example would be helpful…

8

I use the following file structure for my apps

Project_dir
|- server.py
|- app.py
|- layout.py
|- callbacks.py
|- components.py
|- ...
  • server.py contains the creation of app = dash.Dash(). Whenever you need the app, you can import it directly from server.py.
  • callbacks.py contains all the callbacks for my app. (I use @dash.callback() here, because it save me a line on importing app from server.py)
  • layout.py contains the layout of my file and where possible I import components from other files, such as components.py, to keep layout.py as lean as possible (so it’s more easy to see the structure of the layout.
  • app.py: Lastly, this file serves as the file defining your app structure. It imports app from server.py. It imports the callbacks file, so the callbacks are loaded. It imports the layout and assigns it to app.layout. Any other manipulation to the app variable happens here, importing implementations from other files as much as possible to keep the file as clear as possible.

Next you can either put your app.run(debug=True) statement in app.py and run app.py as you main file to start your app. Or you create a different file where you import app from app.py and put the app.run() statement there (this is useful in my context, but will not be necessary for many use-cases).

This setup has solved all circular import issues for me.

P.S. If you need to turn the project directory in a package, just add the file __init__.py to the directory. Leave the file empty. That’s all you need to do.

9

Thank you, this is very helpful.

I would guess that separating callbacks, layout, and component can be done like this: create the components in component.py, import them to callbacks.py where they are manipulated, and then import those manipulated components to layout.py… is that right?

I will be back after adjusting my file structure and providing a more specific example!

10

The more flat file structure works, thank you. My project directory is now essentially the same as above.

However, I have still not achieved my goal of filtering a dataframe [rather than a specific graph]. My use case is as follows:

A switch [eventually a number of switches] that filter the data based on a mask that is applied to the frame. In this case, the feature is “ac”, which has values of either 0 or 1. The default is for the mask to be > -1, and when the switch is turned, the mask becomes > 0.

ac_switch = html.Div([
    html.P("air conditioning"),
    daq.BooleanSwitch(on=False, color="purple", id="ac_bool")
], className="navbar_flex_baby flex_daddy space_between")

the callback is meant to perform this action:

@callback(
    Output(component_id="placeholder", component_property="children"),
    Input(component_id="ac_bool", component_property="on")
)
def update_bool(on):
    if on:
        ac_flag = 0
    else:
        ac_flag = -1

with the output specified as a placeholder and the real change happening to the ac_flag directly. The mask is listed below the callback, though I have tried both above and below to no avail.

mask1 = (df["ac"] > ac_flag)
mask2 = (df["wheelchair"] > wheelchair_flag)
mask3 = (df["parking"] > parking_flag)
mask4 = (df["garden"] > garden_flag)
mask5 = (df["wine"] > wine_flag)
mask6 = (df["terrace"] > terrace_flag)
mask7 = (df["valet"] > valet_flag)
mask8 = (df["vegetarian"] > vegetarian_flag)
mask9 = (df["counter"] > counter_flag)
mask10 = (df["view"] > view_flag)
mask11 = (df["noshoes"] > noshoes_flag)
mask12 = (df["cashonly"] > cashonly_flag)
mask13 = (df["sake"] > sake_flag)

df = df[mask1 
    & mask2 
    & mask3 
    & mask4
    & mask5
    & mask6
    & mask7
    & mask8
    & mask9
    & mask10
    & mask11
    & mask12
    & mask13
]


df_filtered = df

The file structure of this is as below:

Project
| - load_data.py
| - filter_dataframe.py `[just the applicable component and callback]`
| - featureengineering.py
| - components.py
| - index.py
| - server.py
| - app.py

The data imports from the top file to the bottom file in that list, until it reaches index.py, at which point both server and index are imported to app.py, and app.py is run. The app loads fine. But the updated dataframe is not being passed on to the components.

Any ideas on why this callback is not registering?

11

Cant look at your post in detail now, so a small comment. In principle, you dont need to import your components in callbacks.py, unless you specifically need access to the component objects. Manipulating components with callbacks goes via their ids, you dont need the component objects for that.

1 Like
12

Ah, this is useful! Thanks for the comment. Adjusted file structure is below.

| - load_data.py
| - callbacks.py
| - filter_dataframe.py    `[just the applicable ***component***, then exported to index.py]`
| - featureengineering.py
| - components.py
| - index.py
| - server.py
| - app.py

Still no luck in successfully filtering the df using the callbacks described above.

How does one modify a non-dash variable with a callback? Is it even possible?

13

[Updated question]

How does one modify a dataframe with a callback? it can’t be that hard, that’s literally the purpose of a dashboard…

14

@AnnMarieW @Tobs any ideas on this? I will be moving forward without it soon.

15

Of course you can use dataframes in callbacks. I recommend taking a a look at the Dash Fundamentals section of the dash docs.

Plus you can find lots of minimal examples here:

16

Ah phew, that is a relief. I will check out the links you’ve provided. Weird that I am encountering so much difficulty.

Will post an answer later this week if it’s in there.

17

Just a couple of further thoughts - any possibility you could post a short self-contained example that demonstrates the problem?

If that’s not practical the bit that jumps out at me is when you say ‘But the updated dataframe is not being passed on to the components’. I can’t tell from your code how you’re passing the dataframe to the components though - are the components outputs of a callback that contains this code, or are you trying to do it some other way?

Also this bit of code…

df = df[mask1...

… looks like it might be updating a global variable. If it is, that’s generally a bad idea.

18

Hi @davidharris thanks for the response!

In a nutshell, I am curious if the callbacks can work not to update a component, as I understand is the typical use-case, but rather to update a dataframe. In my implementation the callback modifies the mask, which I hoped would happen would then be shown in real time on the front end of the dashboard.

But as you say, the mask is technically global variable. And backed up by my testing indeed it’s not being modified. Damn!

…it sounds like it’s still possible to update a dataframe? The goal is to then feed that updated dataframe to multiple other places in the dashboard. Could one set the callback = a variable as with your standard function, or is that iffy just as the global variable?

Will try and post a true MRE later tonight.

19

The problem with global variables is when multiple users are accessing your dashboard at the same time, and/or when you have multiple agents that can serve incoming requests. The reason is that a global variable is agent-specific and not user-specific. That means the following:

  1. if you have one agent, serving multiple users: changes in the global variable will be changes for all users currently accessing the dashboard
  2. if you have multiple agents, serving any number of users: if your first call, that changes the global variable, is handled by agent 1, that change is only available on agent 1. If you do a second call, and the load balancer of the server passes your request to agent 2 this time, you won’t have access to the changed value on agent 1. Secondly, you can have the issue of point 1, where changes to any of the global variables are applied for all users.

If you need to store user-specific data, the easy way is to use dcc.Store components, that store the data in the browser of the user. This data then needs to be passed along every callback to make it available in the server. Of course, this is not always feasible, for example when dealing with very large dataframes. In that case you will have to use other tricks. Examples of this are available via the pages linked by Ann-Marie.

Lastly, since Dash 2.17.0, there are callbacks with no outputs. This could be convenient if you have a use-case where you don’t want to change anything in your dashboard but need to change something behind the scenes. This comes with a disclaimer: to properly use these, it’s important to get a good grasp on how and where your server will run your tool and how many users you are expecting (see above).

2 Likes
20

Super helpful explanation of global variables and why they don’t work. How do things like JavaScript work, then; ie when one user selects the hamburger menu? That is the level of interaction I am seeking to implement. I suppose I could always dip my toes into JS.

In any case, here’s an MRE:

import numpy as np
import pandas as pd


import dash
import dash_bootstrap_components as dbc
from dash import html
from dash import dcc
import dash_daq as daq
from dash import Input, Output, callback

df = pd.DataFrame({
    "above_equator": [0, 1, 1, 1, 0, 0, 0, 1, 1, 0],
    "right_of_prime_meridian": [1, 0, 1, 1 ,1, 0, 1, 1, 0, 1]
})

ropm_flag = -1
mask =  (df["right_of_prime_meridian"] >  ropm_flag)
df = df[mask]

Placeholder = html.P(";;;", id="placeholder")

Title_card = dbc.Card(
    dbc.CardBody([
        html.H1("this is a div"),
        html.P(f"{len(df)}"),
    ], className="some_css")
)

Switch = html.Div([
        html.P("right of the prime meridian?"),
        daq.BooleanSwitch(on=False, color="blue", id="prime_meridian_boolean")
    ], className="some_other_css")


@callback(
    Output(component_id="placeholder", component_property="children"),
    Input(component_id="prime_meridian_boolean", component_property="on")
)
def update_bool(on):
    global ac_flag
    if on:
        ropm_flag = 0
    else:
        ropm_flag = -1


app_var = dash.Dash(__name__)

app_var.layout = html.Div([
    Title_card,
    Placeholder,
    Switch,
])

server_var = app_var.server

if __name__ == '__main__':
    app_var.run_server(debug=True, 
        # host='0.0.0.0'
        port='8070'
        )

To check if the dataframe is being updated without creating component, there is an f string that prints the length of the frame before and after the switch is pressed. In tandem with the discussion, it does not update.

Any tips on how to update the dataframe much appreciated!