Best method for triggering a backend update/data refresh?

Dash Python
1

I have a multi-page app. I want to have a “refresh data” button on several pages which will call a routine in the backend, wait for it to complete, then refresh the page. The specific routine being called depends on which pages the user is on (and which data will be updated). This seems like it should be an easy process, but I am have having difficultly getting it to work.

Is there a way to just send the browser to the current URL after the call to the external process completes? I don’t think I can set dcc.Location, because then it’ll have multiple input callbacks and that won’t work…

2

You could add an Interval component, which polls the new data e.g. once per second.

3

Thanks for the response, but this won’t work. Please allow me to clarify.

I have a routine which fetches new information from several websites and stores it in a database. This routine runs automatically once per day, for every item. The information displayed in Dash is from the database.

I want to have a button on my front-end Dash app that will trigger this routine manually for the specific item they are looking at, so the information the user is looking at is up to date. Once the routine is completed and the database is updated, the dash webpage should be refreshed, pulling all the new information from the database. If I can just redirect the user to the current page once the routine finishes, then the problem is solved.

My app is set up like this:

index.py - pulls the layout from the appropriate subpage and serves it
app.py - houses the dependencies, and the connection to my backend API

pages/
all.py - shows a summary of all my items. It has a layout variable, which is a navbar, dash datatable, and footer. The navbar and foot are pulled from the components folder.
detail.py - shows a detailed view of the item. it grabs several items from the components folder and assembles them onto the webpage (navbar, footer, summary, history, etc…)

components/
navbar.py - a navigation bar to be used on pages
footer.py - a footer component to be used on pages
summary.py - show a summary of the item, IDs, item name, etc…
history.py - shows a history of transactions for the item
statistics.py - shows different types of statistics for the item for a period defined be the user.

Many of these components already have controls associated with them to manipulate the data, so they can’t have another callback in the output function.

My other thought was to have the button just be a link to a new URL - http://localhost/refresh_data/item/505 would be a page which refreshes the data for item 505, and then redirects them to the previous page, but I think there should be an easier way to do it.

4

I still think that the Interval component would be the way to go. You could do something like,

  • On button click, start the calc on server, set a flag somewhere (in the db, in a memory cache, in a file, …) and start the interval component
  • In the interval component callback, the flag is checked (to see if the calc has completed). If not, do nothing. It if has, disable the interval component and update the page

With the (current) architecture of Dash, you cannot send messages from the server to the client.

5

How does the interval component update the page? The interval’s output cannot be assigned to components that already have callbacks. The page-content output callback is assigned in index.py, and the individual components’ output callbacks are assigned in many cases to controls that manipulate them (e.g. I want to see this data Daily/Monthly/Year/etc).

6

The Interval component works like similar other components, e.g. a button, but it “triggers itself” at a specified interval. Hence, you could probably just add it as an Input to the callback that updates the page content (in index.py).

Depending on your app structure, maintaining the state might require some work. From your statement,

If I can just redirect the user to the current page once the routine finishes, then the problem is solved.

it seems that all state is stored in the url. If this is indeed the case, you should have all state available in the callback that updates the page content already (i assume that the url is an Input? If not so, you can just add it as a State to get the info).

7

I think I understand what your saying now. I think I can just add a input (my_button, n_clicks) to the callback in index.py. Then I can check if the button was clicked in that function and run the update script prior to serving the view. I’m not sure why that didn’t occur to me earlier… I’m not really used to using callbacks, especially not when they are in different files. I think that should work though…

Scheduling a new data pull from a dash interval isn’t necessary, there is already a backend scheduling process that handles this. I’m not sure I explained that well, that might be why I didn’t understand… Unless I’m still not understanding.

8

I spoke too soon, doesn’t work. I tried this:

# index.py
app.layout = html.Div([
    dcc.Location(id='url', refresh=False),
    html.Div(id='page-content'),
    dcc.Interval(id='timer', interval=1, disabled=True),
])


@app.callback(Output('page-content', 'children'),
          [Input('url', 'pathname'),
           Input('refresh-button', 'n_clicks')])
def display_page(pathname, refresh):
print(refresh)
if pathname == '/' or pathname is None:
    return homepage.layout
elif pathname == '/all':
    return all.layout

.

# all.py 
layout = html.Div([
    navbar,
    html.Button(id='refresh-button'),
])

but I get an error:

A nonexistent object was used in an Input of a Dash callback. The id of this object is refresh-button and the property is n_clicks. The string ids in the current layout are: [url, page-content, timer]

The issue is it cannot create a callback for an item that doesn’t exist yet. In some cases, this item (the refresh data button) may not exist at all because some pages have no data to refresh.

I’m still not sure I understand how the Interval control is supposed to fix this. Are you suggesting that the interval control refreshes the data every x seconds?

9

Could signaling like in https://dash.plotly.com/sharing-data-between-callbacks Example 3 - Caching and Signaling work for you? The basic idea is that a button click triggers a callback which runs your routine. When the routine is complete, the callback outputs a ‘signal’ to a property of a hidden div. Every callback handling the data shown to the user should have a this hidden div property as an input so that the data obtained from the db gets updated when the signal is triggered.

10

No, my intention was to trigger the page refresh by the Interval component. So more like (pseudo code),

@app.callback(Output('page-content', 'children'),
          [Input('url', 'pathname'),
           Input('timer', 'n_intervals')])
def display_page(pathname, n_intervals):
trigger_id = get_trigger_id()  # get trigger id somehow
if trigger_id == 'timer':
    refresh_needed = is_refresh_needed() # check if refresh is needed somehow
    if not refresh_needed:
        raise dash.Preventupdate
    # The data are ready. Wuhuu! Proceed to update the page normally
if pathname == '/' or pathname is None:
    return homepage.layout
elif pathname == '/all':
    return all.layout
11

@Emil the problem is, how do I determine is_refresh_needed()? I can’t assign a callback to a control that doesn’t exist. And if I could, I wouldn’t need the interval timer, I could just call the routine directly. This is what i keep running into.

@psip I looked at that yesterday for a little while, but I think I’m going to run into the same issue as above.

My entry point is index.py, which I think that should be fetching the data from the db and sending it to the page requiring it (all.py, homepage.py, etc) or sending an injected service container to the page so it can query the information itself (this is how it’s done now).

The page files then load the individual components - summary info, history, statistics, etc. The components are meant to be reused throughout the app. I want them to be modular so i feed them the data and they return the HTML for the component, plus any controls needed to manipulate the component. The page just puts all these components together and arranges them to look nice.

The problem with this design is that a page isn’t going to have all the same controls. I have no way passing data up from a component to the main “controller” (index.py). So if I have a button called “refresh-data”, I can’t assign it’s callbacks to a “storage” object on the “parent/entry” level, because buttons on other pages are going to want to write to that same object, and multiple objects can’t have the same output target.

Right now app.layout is being assigned in index.py, and I’m wondering if each page should have an app.layout instead…

12

The is_refresh_needed reads a server side flag (from a file, a memory cache, the database itself). This flag should set by the async routines that fetches the data.

13

@Emil that might work, but it seems like an inelegant solution. It seems like it’s going to create dependencies where there shouldn’t be any. I’m not sure where exactly, but something about it feels wrong from an object oriented perspective. It’s forcing the lower level code to know about higher level problems, which will inevitably lead to tight coupling.

14

It should be possible to do an implementation without introducing weird dependencies. There are many possible routes, which one is the best will ultimately depend on your remaining architecture. Here are two options,

  • You could introduce a variable in the database that keeps track of the database state. A simple option would be a last_edit_timestamp (which might already be available depending on the db, if not it should be simple to introduce it via a trigger). Since this variable is essentially meta data about the data, i see no problem in holding it inside the db from an object perspective. If you don’t like putting the data in the db, setup a Redis cache (probably the fastest option) or write it to a file instead.

If you choose this option, the app should fetch the last_edit_timestamp along with the data on initial load and cache it. The is_refresh_needed function would then fetch the current last_edit_timestamp from the db and return True if it is newer than the last_edit_timestamp in the clients cache.

  • You could run the async job via a task queue, e.g. Celery, using a broker (such as rabbit mq) for communication. On task completion, you could then refresh the page as suggested above, or you could pass relevant data via a memory cache, e.g. Redis.

If you choose this option, the is_refresh_needed function will essentially just be the ready() method of on the result` object.

15

I got this working a different way using javascript and visdcc.

# page.py
import visdcc

# controls
visdcc.Run_js('javascript'),
html.Button('Refresh Page', id='refresh-page')

# callback
@app.callback(
    Output('javascript', 'run'),
    [Input('refresh-page', 'n_clicks')])
def refresh_button_clicked(data):
    if data is not None:
        fetch_new_data_function()
        return "location.reload();"
    return ''

Thanks for your help!

1 Like
16

Is the data fetch sync or async?

As I read your solution, a sync data fetch would result in the GUI freezing until the data fetch completes. This is not so nice from a UX perspective, but might be OK depending on the application.

An async data fetch on the other hand would result in a correct update of the page if the data fetch is fast enough that it completes before the client request arrives. Hence you are essentially introducing a race condition. If the data fetch is guaranteed to be fast enough, this will work consistently, but I would generally avoid against this type of architecture as it might introduce weird bugs if the timing changes later on.

17

Try this: Output(‘url’, ‘pathname’) as an additional output for your callback. Get the current URL using State(), and simply serve it back (unchanged) at the end of your callback. I haven’t tried it, but it should “refresh” the page by triggering your (separate) navigation callback which has Input(‘url’, ‘pathname’) as a trigger.

Gosh I love that we now support multiple outputs.

1 Like
18

Hi @mathybit, can you give a complete example?

1 Like
19

This worked for me, thanks!
And it is a pretty neat way too :slight_smile:

20

Hey @mathybit @Math
I’m new to dash and I didn’t really understand what you’ve written. Could you please give an example.