How to integrate dash with another process that might be blocking

Dash Python
1

This is somewhat related to the aim described in this post. I want to use dash to control a piece of equipment. User sets some parameters, sends the command to the equipment and then gets some data back to monitor variables of interest in real time.

Maybe, for example, we start up dash and then send a command to a hypothetical wireless coffee maker. The API to the coffee might have a brew() call that begins the heating process, adding it’s temperatures to a data frame so we can watch the process. Perhaps brew() doesn’t return until the coffee maker reaches a target temp.

Here’s an example illustrating what I mean:

#!/usr/bin/env python

import dash
import dash_core_components as dcc
import dash_html_components as html
import datetime
import numpy as np
import pandas as pd
import plotly.graph_objs as go
import time


global data
data = pd.DataFrame(columns=['time', 'temp'])


def build_layout():

    layout = html.Div([
        html.H2('a dashboard'),
        html.Div(dcc.Interval(id='refresh', interval=1000)),
        html.Div(id='plot')
    ]) # app.layout

    return layout


def generate_plot(data):
    
    plot = dcc.Graph(
        id='the_plot',
        figure={
            'data': [go.Scatter(name='plot',
                                x=data['time'],
                                y=data['temp'])],
            'layout': go.Layout(title= 'Live test data')#,
        },
        style={'width': '80%', 'float': 'right'}
    ) # dcc.Graph

    return plot


def generate_data():
    count = 0
    while True:
        for i in range(10):
            data.loc[len(data)] = [datetime.datetime.now(), np.random.uniform(50)]
            print(data)
            time.sleep(0.5)


app = dash.Dash()
app.layout = build_layout


@app.callback(
    dash.dependencies.Output('plot', 'children'),
    events=[dash.dependencies.Event('refresh', 'interval')])
def update():
    return generate_plot(data)


if __name__ == '__main__':

    app.run_server()
    generate_data()

So we set up the dashboard (just a plot) and want it to update every second. I know that global variables are not recommended, but I’m a noob so it’s the best I could figure at the moment. After starting the server, generate_data() is like the brew() call mentioned above. It doesn’t return until brewing is complete, but it’s updating data at regular intervals.

The example above behaves like run_server() is blocking somehow. It starts up, but I get a blank plot, and data is never printed. When I Ctrl-C in the terminal, the data starts printing out as expected.

In this example, I can change the callback to (adding a return to generate_data()):

def update():
    data = generate_data()
    return generate_plot()

And that will update the plot at 1 sec intervals as expected. But… I don’t have the analog of this in the method I described.

How do you blend dash with something that blocks or doesn’t return when you want? Do I need to separate these and, say, have one python file/thread writing interim data to a file, and a separate dash process where the callback would read that file? I hoped to avoid this so everything could just pass a data object around vs. having to rely on I/O.

Thanks for any tips!

2

That’s what I would recommend doing. Create a semaphore and write it to the disk. IO is great because it works across processes. Here are some things to consider (all of which should be handled by using IO):

  • What happens when multiple people visit the dash app to control the hardware? When reading from a file, each new session will get its “source of truth” from the file.
  • What happens when you deploy across multiple processes? Right now, a single callback / request can block the entire process (preventing any other callbacks or requests from firing). To solve this, you’ll need to run this app on multiple processes/threads with something like $ gunicorn app:server --workers 4 --threads 2. In this case, the memory won’t be shared across processes which is why modifying global variables isn’t safe. However, reading files is.

Here’s a simple example with a file lock:

dash-asynchronous
dash-asynchronous837×649 1.41 MB

import dash
from dash.dependencies import Input, Output, Event
import dash_core_components as dcc
import dash_html_components as html

import datetime
import time


class Semaphore:
    def __init__(self, filename='semaphore.txt'):
        self.filename = filename
        with open(self.filename, 'w') as f:
            f.write('done')

    def lock(self):
        with open(self.filename, 'w') as f:
            f.write('working')

    def unlock(self):
        with open(self.filename, 'w') as f:
            f.write('done')

    def is_locked(self):
        return open(self.filename, 'r').read() == 'working'


semaphore = Semaphore()


def long_process():
    if semaphore.is_locked():
        raise Exception('Resource is locked')
    semaphore.lock()
    time.sleep(7)
    semaphore.unlock()
    return datetime.datetime.now()


app = dash.Dash()
server = app.server


def layout():
    return html.Div([
        html.Button('Run Process', id='button'),
        dcc.Interval(id='interval', interval=500),
        html.Div(id='lock'),
        html.Div(id='output'),
    ])


app.layout = layout


@app.callback(
    Output('lock', 'children'),
    events=[Event('interval', 'interval')])
def display_status():
    return 'Running...' if semaphore.is_locked() else 'Free'


@app.callback(
    Output('output', 'children'),
    events=[Event('button', 'click')])
def run_process():
    return 'Finished at {}'.format(long_process())


if __name__ == '__main__':
    app.run_server(debug=True, processes=5)

Let me know if that helps!

1 Like
3

A post was split to a new topic: Update CSV From A Different Script

4

Many moons later and I’m back to a need for this! Sorry for the delay, and I’m still finding this to be an issue. I think the issue lies in spinning up something else that is also watching/reacting/etc. from Dash. I think this post is similar and my need, I think, is bigger than just writing a status to a file.

I’m trying to make a user interface for some Robot Operating System (ROS) code. ROS enables many nodes to co-exist and communicate on things called topics. An analogy might be something like a webserver that can do GET/POST. I’ve been hoping to combine these things into one, but am having a tough time.

So imagine you have some other library and need to create an object listening/publishing on some other port… maybe:

foo = Server(port)
foo.start()

In addition, perhaps periodically you do foo.publish().

What I’m finding is if I initialize this up with app = dash.Dash(__name__), it recreates with any page refresh, so I get errors due to the node already existing. I can put some checks in a function like your example, but I don’t know how to pass foo around to run foo.publish().

Do you have recommendations of where to create some object you need to interact with, (assuming that this server analogy makes sense and is possible with Dash)?

Also, for a secondary process, how does one kill off everything when you kill Dash?

5

If you just want to run tasks in a (single) separate process, I’d recommend checking out Celery. We have an example here: GitHub - plotly/dash-redis-demo: Connect to Redis from Dash

1 Like
6

Just an update if someone stumbles on this. I ended up ditching this concept and separating Dash from the ROS infrastructure. It might be solvable, but I kind of gave up troubleshooting. I think the issue of having Dash be a ROS node is there are two things sitting there watching for updates. Might be wrong.

In the end, I use Dash to queue up the right message type and then write them via rosbag to file. Then I have an actual ROS node watching a jobs/ directory and taking care of communication with ROS to process them.

I’m currently using some simple status.txt file to let the GUI know the status of things (startup, idle, processing, etc.).

Anyway, the file system is pretty flexible as it turns out :slight_smile: Thanks for the coaching/assistance, @chriddyp !

2 Likes
7

Update from a lurker. I manage to plot dynamically from a constantly updating text file. if anyone wants my code just shout out. I basically used

  • a .txt in which I log the latest values in a data string
  • a callback triggered by dcc.Interval that reads the values and sticks them in a dcc.Store
  • another callback triggered by the same dcc.Interval that gets the values from dcc.Store into a np.array and plots them

This should be useful for ROS too, using a rosbag log perhaps.