How to plot multiple .csv files in a single graph on a dashboard

Hi All,

I have a dropdown-driven dashboard for plotting csv file data. Apart from individual cvs plots, I have to plot a trend chart combined for all the .csv files present in a directory(using only the selected columns for the .csv files. Note: All csv files have same columns, different values.) using plotly express, and add it to this same dashboard. Plotting them and then integrating with the existing dashboard is where I am stuck.

The code has two dropdown,

  • First is multi-selection: For selecting the files to be plotted
  • Second is single-selection: For column to be considered for plotting

Below is my sample code:-

import dash
from dash import dcc, Input, Output, html, dash_table
import plotly.express as px
import pandas as pd
import os
# import plotly
import plotly.graph_objects as go


# ======================== Dash App
app = dash.Dash( __name__ )

# ======================== Getting input of directory and Latest filename
PATH = str( input( "Enter Full file path" ) )  # Use your path
# LatestFile = str( input( "Enter Full file Name" ) )  # Use your latest filename

### Fetch all files in path
fileNames = os.listdir( PATH )

### Filter file name list for files ending with .csv
fileNames = [file for file in fileNames if '.csv' in file]

print( fileNames )


# ======================== App Layout
app.layout = html.Div( [
    html.H1( 'Trend Analytics Dashboard', style={'text-align': 'center', 'background-color': '#ede9e8'} ),
    html.Div( [
            dcc.Dropdown( id='DropDown_FileName',
                          options=[
                              {'label': i, 'value': i} for i in fileNames
                          ],
                          # value=fileNames,
                          placeholder='Select Files for Trend Analysis',
                          multi=True,
                          clearable=False ),
                    ] ),
    html.Div( [
        dcc.RadioItems( id='DropDown_Value',
                      options=[
                          {'label': 'Average', 'value': 'Average'},  {'label': 'Samples', 'value': 'Samples'},
                          {'label': 'Throughput', 'value': 'Throughput'}, {'label': 'Comb_wgt', 'value': 'Comb_wgt'}
                      ],
                      value='Average',
                      #placeholder='Select a value',
                      # multi=False,
                      # clearable=False
                      ),
            ] ),

    html.Div( [
        dcc.Graph( id='TrendChart' ),
    ] ),


] )


@app.callback(
    Output( 'TrendChart', 'figure' ),
    Input( 'DropDown_FileName', 'value' ), Input( 'DropDown_Value', 'value' )
)
def update_figure(DropDown_FileName, DropDown_Value):
    print(DropDown_FileName)
    print(type(DropDown_FileName))
    for file in DropDown_FileName:
        df_comb = pd.read_csv(PATH + file)
        df_comb['Comb_wgt'] = df_comb.Samples * df_comb.Average
        df_comb.rename( columns={'Label': 'Request'}, inplace=True )
        df_comb = (df_comb.sort_values( by='Request', ascending=True ))
        df_comb['Average'] = df_comb['Average']/1000
        print(df_comb)
        TrendChart = px.bar(data_frame=df_comb, x=df_comb['Request'], y=df_comb[DropDown_Value],
            color='Request',
            title=" Trend Analysis Graph by"+DropDown_Value,
            text=df_comb[DropDown_Value],
            labels={'Request': 'Request', DropDown_Value: DropDown_Value},
        ).update_traces(textposition='outside')

        return TrendChart


if __name__ == '__main__':
    app.run_server()

With the above code, the for loop gets stuck after the first csv file itself. Graph is plotted for just the first file only, while other files are not even read. I checked the return type for “DropDown_FileName”, and its coming as list, also the radio selection is working fine.
Somehow the for loop is not working properly and the graph is not getting updated.

Objective is to have data from all csv file present on the same graph for trend analysis that is to say, just update the graph without deleting the previous plots

Thank you in advance.

Does this help?

Hi @AIMPED,

Thank you for an early reply.
I have modified my code and have added two dropdowns, one as multiple selection=True and other as multiple selection=False or single selection only.

I checked the link you shared, and took the same approach as suggested in the link by modifying my callback function as below:

@app.callback([
    Output( 'TrendChart', 'extendData' )],
    [Input( 'DropDown_FileName', 'value' ), Input( 'DropDown_Value', 'value' )]

Both mine and the one using above callback, make the code get stuck post reading first csv file. Also no graph is getting plotted.
Can we look into the code so as to find what I am missing !

Hi @atharvakatre, @chriddyp, @alex_01, @AIMPED

Guys, I have looped you people in for my question as in many queries on plotly community, I could see you all have given effective solutions. Kindly go through the thread and please let me know the solution.

I have updated code as below:
I changed the3 indentation of return statement.

With above adjusting the return statement of my function, I am able to get the plot for the last file that user selects from the dropdown. For previous file selection, no plotting is preserved, data is lost. Plot shows just the data from last selected file only.

Also I used “Dash-extendable-graph” with “extendData” as output type. Only the last file gets read from the selection, but no graph is plotted even for that file data.

The actual problem is that previous data is not preserved and with each new file that is selected, the plot reflects only the latest data on graph.

Please look into this.

Hi @brad, @Mike3 @markm @tubuliferous @tomazmalika

Could you guys please look into the issue.
Basically here I am trying to plot all the .csv files selected through dropdown on the same graph. Plotted columns are same for all the .csv files.

Thank you

@Himanshu_Shukla,

Looking at your code, it looks like instead of combining the charts into one based upon the data, you are creating a chart for the first file selected and then returning it, that’s why it looks like it is getting stuck.

Can you combine the separate csv files into one df and then pass that df to make the bar?

Hi @jinnyzor

Thank you for responding, been stuck at this since a long time!!!
I cannot combine different .csv files into a single df, as these .csv files have the same requests for which just the average, samples, throughput etc values is changing.

So when I try to combine all into one df, each product values like Average/sample/throughput/…ect gets added up. And a collective chart for each product gets plotted.
What I want is, to show a trend for each product across various .csv files.
Please do let me know if I need to give some more info, as I badly need the a solution to this.

Regards,
Himanshu

@Himanshu_Shukla,

Is it possible to use a box plot for these instead of a bar chart? That would allow for the multiple values to be added together as separate plots instead of just added together as a total. To plot these, you would add the dataframes together and use the final dataframe to plot the box.

Then for the total of your selection at the top [‘Average’.‘Samples’,‘Throughput’,‘Comb_wgt’] you can use cards with information that you want to display overall, instead of skewing your data and making it hard to read as the total will exponentially grow as you add more files.

Not sure if this will work, as I dont have your data:

import dash
from dash import dcc, Input, Output, html, dash_table
import plotly.express as px
import pandas as pd
import os
# import plotly
import plotly.graph_objects as go


# ======================== Dash App
app = dash.Dash( __name__ )

# ======================== Getting input of directory and Latest filename
PATH = str( input( "Enter Full file path" ) )  # Use your path
# LatestFile = str( input( "Enter Full file Name" ) )  # Use your latest filename

### Fetch all files in path
fileNames = os.listdir( PATH )

### Filter file name list for files ending with .csv
fileNames = [file for file in fileNames if '.csv' in file]

print( fileNames )


# ======================== App Layout
app.layout = html.Div( [
    html.H1( 'Trend Analytics Dashboard', style={'text-align': 'center', 'background-color': '#ede9e8'} ),
    html.Div( [
            dcc.Dropdown( id='DropDown_FileName',
                          options=[
                              {'label': i, 'value': i} for i in fileNames
                          ],
                          # value=fileNames,
                          placeholder='Select Files for Trend Analysis',
                          multi=True,
                          clearable=False ),
                    ] ),
    html.Div( [
        dcc.RadioItems( id='DropDown_Value',
                      options=[
                          {'label': 'Average', 'value': 'Average'},  {'label': 'Samples', 'value': 'Samples'},
                          {'label': 'Throughput', 'value': 'Throughput'}, {'label': 'Comb_wgt', 'value': 'Comb_wgt'}
                      ],
                      value='Average',
                      #placeholder='Select a value',
                      # multi=False,
                      # clearable=False
                      ),
            ] ),

    html.Div( [
        dcc.Graph( id='TrendChart' ),
    ] ),


] )


@app.callback(
    Output( 'TrendChart', 'figure' ),
    Input( 'DropDown_FileName', 'value' ), Input( 'DropDown_Value', 'value' )
)
def update_figure(DropDown_FileName, DropDown_Value):
    print(DropDown_FileName)
    print(type(DropDown_FileName))
    df_comb = pd.DataFrame()
    for file in DropDown_FileName:
        df = pd.read_csv(PATH + file)
        df['Comb_wgt'] = df.Samples * df.Average
        df.rename( columns={'Label': 'Request'}, inplace=True )
        df = (df.sort_values( by='Request', ascending=True ))
        df['Average'] = df['Average']/1000
        print(df)
        df_comb = pd.concat([df_comb, df])
    TrendChart = px.box(data_frame=df_comb, x=df_comb['Request'], y=df_comb[DropDown_Value],
        color='Request',
        title=" Trend Analysis Graph by "+DropDown_Value,
        labels={'Request': 'Request', DropDown_Value: DropDown_Value},
    ).update_traces(textposition='outside')

    return TrendChart


if __name__ == '__main__':
    app.run_server()

Hello @jinnyzor,

many thanks for the efforts and time. I greatly appreciate it. :slight_smile:

Using the box plot, I am able to plot the .csv file data, but actually the need is to have either line chart, or bar graph for the requests.
Please find below example:

The above graph has been plotted using excel. Here I have considered the averages across 3 .csv files. This is how I want to plot the data so as to better reflect the trend even to a person with quite less knowledge towards datas and graph.
Box plot still requires some statics info for understanding. Box plot gets more and more complex with more number of .csv data files.

Can we look into something like the graph shared.

Please do let me know if any more detail is required.

@Himanshu_Shukla,

Try this:

import dash
from dash import dcc, Input, Output, html, dash_table
import plotly.express as px
import pandas as pd
import os
# import plotly
import plotly.graph_objects as go


# ======================== Dash App
app = dash.Dash( __name__ )

# ======================== Getting input of directory and Latest filename
PATH = str( input( "Enter Full file path" ) )  # Use your path
# LatestFile = str( input( "Enter Full file Name" ) )  # Use your latest filename

### Fetch all files in path
fileNames = os.listdir( PATH )

### Filter file name list for files ending with .csv
fileNames = [file for file in fileNames if '.csv' in file]

print( fileNames )


# ======================== App Layout
app.layout = html.Div( [
    html.H1( 'Trend Analytics Dashboard', style={'text-align': 'center', 'background-color': '#ede9e8'} ),
    html.Div( [
            dcc.Dropdown( id='DropDown_FileName',
                          options=[
                              {'label': i, 'value': i} for i in fileNames
                          ],
                          # value=fileNames,
                          placeholder='Select Files for Trend Analysis',
                          multi=True,
                          clearable=False ),
                    ] ),
    html.Div( [
        dcc.RadioItems( id='DropDown_Value',
                      options=[
                          {'label': 'Average', 'value': 'Average'},  {'label': 'Samples', 'value': 'Samples'},
                          {'label': 'Throughput', 'value': 'Throughput'}, {'label': 'Comb_wgt', 'value': 'Comb_wgt'}
                      ],
                      value='Average',
                      #placeholder='Select a value',
                      # multi=False,
                      # clearable=False
                      ),
            ] ),

    html.Div( [
        dcc.Graph( id='TrendChart' ),
    ] ),


] )


@app.callback(
    Output( 'TrendChart', 'figure' ),
    Input( 'DropDown_FileName', 'value' ),
    Input( 'DropDown_Value', 'value' ), prevent_initial_call=True
)
def update_figure(DropDown_FileName, DropDown_Value):
    df_comb = pd.DataFrame()
    for file in DropDown_FileName:
        df = pd.read_csv(file)
        df['file'] = file
        df['Comb_wgt'] = df.Samples * df.Average
        df.rename( columns={'Label': 'Request'}, inplace=True )
        df = (df.sort_values( by='Request', ascending=True ))
        df['Average'] = df['Average']/1000
        df_comb = pd.concat([df_comb, df])
    TrendChart = px.bar(data_frame=df_comb, x=df_comb['Request'], y=df_comb[DropDown_Value],
                        barmode='group',
        color='file',
        title=" Trend Analysis Graph by "+DropDown_Value,
        labels={'Request': 'Request', DropDown_Value: DropDown_Value},
    ).update_traces(textposition='outside')

    return TrendChart


if __name__ == '__main__':
    app.run_server(port=8054, debug=True)

Hi @jinnyzor,

When I used the above code shared by you(exactly the same), I am getting the below error on UI:


There is no error on the console. I am using PyCharm.

When I am using below code which just has same function as you shared, but not in DEBUG mode

if __name__ == '__main__':
    app.run_server()

then I am getting below error:
*FileNotFoundError: [Errno 2] No such file or directory: *
‘CockPit_N01_14VU_NONLabel_02Agst_aggregate.csv’
127.0.0.1 - - [05/Oct/2022 00:18:02] “POST /_dash-update-component HTTP/1.1” 500 -

Though, the file is present in the folder, since it is getting populated in the dropdown.

@jinnyzor The graph you shared is exactly what is required for me, It would really be great if you could just guide me with this.

I would be glad to connect, if possible.

Regards,
Himanshu

Change this

df = pd.read_csv(file)

To this

df = pd.read_csv(PATH + file)

I didnt reset this line.

1 Like

Hi @jinnyzor

Many thanks!!!
This is the required solution to the problem that I was facing.

barmode='group',

“barmode” did actually work. Thank you once again.

Regards,
Himanshu

Hi @jinnyzor

I wanted to include the user input part of my app in the app layout as well.
Currently for directory, the user inputs the complete path on the console but I want to include this input entering part on the UI itself.

PATH = str( input( "Enter Full file path" ) )  # Use your path
# LatestFile = str( input( "Enter Full file Name" ) )  # Use your latest filename

### Fetch all files in path
fileNames = os.listdir( PATH )

### Filter file name list for files ending with .csv
fileNames = [file for file in fileNames if '.csv' in file]

Above code, I want to have as part of UI as well, where the user would enter directory path and then .csv files for that directory would get populated in the drowdowns below used.

I am unable to do it and pass the user given directory path to the below dropdowns.

Can you please look into this.

Regards,
Himanshu