Saving a CSV that is continuously updating with data

Dash Python
1

Hello there,

Context:

I am currently working with a Programmable Logic controller (PLC) and using Modbus to read values from Analog Inputs.
With these values, I want to introduce them into graphics so that I can see the dynamics of my system.
I would also like to save these values to a CSV file using the open and csv objects from python.
This last part is where it gets tricky, I’ll show you my simplified code where I am not reading values but instead randomly generating them (x, y, z).

Code:

import dash
import dash_html_components as html
import dash_core_components as dcc
from dash.dependencies import Input, Output, State
from random import random
import csv

x, y, z = 0, 0, 0
save_list = [str(x), str(y), str(z)]

app = dash.Dash()

app.layout = html.Div([
     html.Div([
        html.Button('Save', id='button-save', n_clicks=0),
        html.Button('Stop', id='button-stop', n_clicks=0),
        dcc.Store(id='save-status', data=False),
        html.Div(id='save-csv'),
        dcc.Interval(
            id='interval-component',
            interval=2000,    # 1000 miliseconds
            n_intervals=0
            )
    ])])

@app.callback([Output('save_status', 'data'),
               Output('save_csv', 'children')],
             [State('button-save', 'n_clicks'),
              State('button-stop', 'n_clicks'), 
              State('save_status', 'data')])
def Save_stop(n_clicks3, n_clicks4, save):
    global save_list, write_csv
    changed_id_2 = [p['prop_id'] for p in dash.callback_context.triggered][0]
    if 'button-save' in changed_id_2:
        save = True
        with open('Some/path/to/file.csv.csv', 'w') as writer:
            write_csv = csv.writer(writer, delimiter='\n')
            write_csv.writerow(save_list)
        return save, html.Plaintext('{}'.format(save))
    elif 'button-stop' in changed_id_2:
        save = False
        return save, html.Plaintext('{}'.format(save))

@app.callback(Output('save_status', 'data'),
             [Input('interval-component', 'n_intervals')],
             [State('save_status', 'data')])
def Build_csv(n, save):
    global x, y, z, write_csv
    if save:
        x, y, z = random(), random(), random()
        temp_line = [str(x), str(y), str(z)]
        with open('Some/path/to/file.csv', 'w') as writer:
            write_csv = csv.writer(writer, delimiter='\n')
            write_csv.writerow(temp_line)
    return save


if __name__ == '__main__':
    app.run_server(debug=True)

Further thoughts:

What I am doing is using one callback to set “writting mode” to True or False, create the CSV file and write the index in case of True, and the other callback to keep adding to the previously generated CSV file.

Basically nothing happens, the file isn’t generated anymore, but for attempts where it was created, nothing was written. And I can’t close it.

I am a bit cluesless as of what to try more. Is there anything wrong with the callbacks? Should I not use the withstatement in this case since I am trying to open it in both callbacks? Also, the path to file does not have blank spaces.

Thank you all in advance.

2

Hi,

Regarding your callbacks, I can see two problems that will impact your app:

  1. Both callbacks have the same output: Output("save_status", "data"). I am quite sure this should have raised an error in debug mode, but maybe I am wrong… In any case, you are not allowed by default to have duplicated outputs.

  2. The Save_stop callback does not have a Input, therefore it is only triggered when the application starts.

The solution to both problems is to combine the callbacks in one and use the context to figure out what actions to do:

If it was triggered by Interval, then you save the data.

If it was triggered by a button, you can enable/disable Interval using the prop “disabled” (see the docs for it), which will prevent/enable interval to trigger updates.

From an UX perspective, it is not optimal to have two buttons to control a single behavior. A checkbox or toggle would be better. Besides, it is not recommended to use global variables in Dash, their behavior not what you expect due to the “distributed” nature of callbacks.

3

I changed my code to this:

import dash
import dash_html_components as html
import dash_core_components as dcc
from dash.dependencies import Input, Output, State
from random import random
import csv
import dash_daq as daq

app = dash.Dash()

app.layout = html.Div([
        daq.ToggleSwitch(
            id='my-toggle-switch',
            value=False),
        html.Div(id='save-csv'),
        dcc.Interval(
            id='interval-component',
            interval=1000,    # 1000 miliseconds
            n_intervals=0
            )
    ])

@app.callback(Output('save-csv', 'children'),
              [Input('interval-component', 'n_intervals'),
               Input('my-toggle-switch', 'value')])
def Create_csv(n, value):
    changed_id = [p['prop_id'] for p in dash.callback_context.triggered][0]
    if value:
        if 'my-toggle-switch' in changed_id:
            x, y, z = 'index-1', 'index-2', 'index-3'
            temp_line = [x, y, z]
            with open('Some/path/to/file.csv', 'w') as writer:
                write_csv = csv.writer(writer, delimiter=',')
                write_csv.writerow(temp_line)
        elif 'interval-component' in changed_id:
            x, y, z = random(), random(), random()
            temp_line = [str(x), str(y), str(z)]
            with open('Some/path/to/file.csv', 'w') as writer:
                write_csv = csv.writer(writer, delimiter=',')
                write_csv.writerow(temp_line)
    return html.Div('Saving: {}, changed_id: {}'.format(value, changed_id))


if __name__ == '__main__':
    app.run_server(debug=True)

Some things:

I joined both call backs and now I can create a CSV file, but it doesn’t start with the indexes: x, y, z = 'index-1', 'index-2', 'index-3', ig goes straight to the random numbers: x, y, z = random(), random(), random()

Shouldn’t the order of things be like:

  • If toggle becomes True, the changed_id becomes 'my-toggle-switch.value' and it writes the indexes

  • The next iterations will be triggered by 'interval-component.n_intervals' and new random values will be added until the toggle becomes False again.

Also, it only writes once the random values and never again. The iterations proceed though, because changed_id updates in return html.Div('Saving: {}, changed_id: {}'.format(value, changed_id)).

I tried disabling the interval dcc but if I disable it, there won’t be any trigger to keep saving data and it won’t enable automatically. So i think I didn’t get the idea and perhaps I don’t fully understand how it works.

4

Ok, turns out the indexes are written after all.

What seems to happen is that the code generates a new CSV file from scratch everytime the inputs are triggered, so everytime I open the CSV file to check what’s in it, I only see the latest random values generated.

I’ll work on it and give feedback later. Help is appretiated of course!

5

I think you have to open the file in append mode “a”, instead of “w”.

Glad you figured out by the way! I had something slightly different in mind, but I am on mobile and it would be impossible to write proper code.

1 Like
6

Thank you @jlfsjunior, turns out the append mode “a” was the missing piece!

Here is the final snippet for those who need it:

import dash
import dash_html_components as html
import dash_core_components as dcc
from dash.dependencies import Input, Output, State
from random import random
import csv
import dash_daq as daq

app = dash.Dash()

app.layout = html.Div([
        daq.ToggleSwitch(
            id='my-toggle-switch',
            value=False),
        html.Div(id='save-csv'),
        dcc.Interval(
            id='interval-component',
            interval=1000,    # 1000 miliseconds
            n_intervals=0
            )
    ])

@app.callback(Output('save-csv', 'children'),
              [Input('interval-component', 'n_intervals'),
               Input('my-toggle-switch', 'value')])
def Create_csv(n, value):
    changed_id = [p['prop_id'] for p in dash.callback_context.triggered][0]
    if value:
        if 'my-toggle-switch' in changed_id:
            x, y, z = 'index-1', 'index-2', 'index-3'
            temp_line = [x, y, z]
            with open('Some/path/to/file.csv', 'w', newline='') as writer:
                write_csv = csv.writer(writer, delimiter=',')
                write_csv.writerow(temp_line)
        elif 'interval-component' in changed_id:
            x, y, z = random(), random(), random()
            temp_line = [str(x), str(y), str(z)]
            with open('Some/path/to/file.csv', 'a', newline='') as writer:
                write_csv = csv.writer(writer, delimiter=',')
                write_csv.writerow(temp_line)
    return html.Div('Saving: {}, changed_id: {}'.format(value, changed_id))


if __name__ == '__main__':
    app.run_server(debug=True)

Cheers!

7

Result:

result-file
result-file965×728 42.4 KB

8

I am glad you found a solution. I just wanted to mention a technique I have used for collecting and plotting live data.
It only works in a situation where your app will only be used locally but for sensor data I think that is usually the case. basically I use a background thread to collect data over the serial port, process it and send to a queue object and eventually to a sqlite db file instead of a csv file. I found the sqlite db file a little more stable than csv especially when the data gets pretty big. the added benefit is you can use a pretty simple sql queries to pull evenly spaced data with a max number of data points to display. In some long running tests I found the large number of data points overloaded the browser.

You can check out a working example here:
shkiefer/DAQ-capture-serial: Python web-app for capturing json-like data over serial. (github.com)

May be more than needed for your application but maybe it will help someone else.

2 Likes