Update options in the dropdown list

Dash Python
1

I have the following code that reads files and extract the input for the dropdown option list, then there is the layout and callback to plot some graphs, the options definition is set before the layout so the layout can find the variables needed. However the list is supposed to be updating every 10 seconds, but because I have created the options list outside the layout, it does not get updated, and any new information is not coming to the list.
The question is that what can I do to have it updated along with the update of the graph? I moved the lists creation inside the callback but it fails as the layout will not find these options. I also duplicated the same and put another copy inside the callback, but it did not help, the list is still based on the initial files reading.

files = glob.glob("C:/DataCapture/*.txt")
modified_files = list()
current_time = time.time()
for txt_file in files:
    time_delta = current_time - os.path.getmtime(txt_file)
    if time_delta < 90:
        modified_files.append(txt_file)
dfs = [ ]
for file_name in modified_files:
    df1 = pd.read_csv(file_name, on_bad_lines='skip').dropna()
    df1.rename(
    columns={'_ws.col.Time': 'Time', 'ip.src': 'Source', 'ip.dst': 'Destination', 'ip.proto': 'Protocol'},
            inplace=True)
    dfs.append(df1)
df = pd.concat(dfs)
df['Protocol'] = df['Protocol'].replace({1: 'ICMP', 2: 'IGMP', 6: 'TCP', 17: 'UDP', 80: 'Http', 443: 'Https'})
list_src_IP = list(df['Source'].unique())
list_dst_IP = list(df['Destination'].unique())
list_prt = list(df['Protocol'].unique())

layout = html.Div([
    html.H1('You can enter your filtering options below', className='text-center fs-1 text-primary'),
    dcc.Interval(
        id = 'Interval',
        interval = 5 *2000,
        n_intervals = 0
        ),
    dcc.Dropdown(
        id='src_IP_options',
        placeholder = 'Select The Source IP or leave empty',
        options = list_src_IP,
        style={"width": "60%"}),
    dcc.Dropdown(
        id='dst_IP_options',
        placeholder = 'Select The Destination IP or leave empty',
        options = list_dst_IP,
        style={"width": "60%"}),
    dcc.Dropdown(
        id='prt_options',
        placeholder = 'Select The Port or leave empty',
        options = list_prt,
        style={"width": "60%"}
        ),
    html.Br(),
    html.Label(['']),
    html.Button(id='submit_btn', n_clicks=0, children='Submit'),
    html.Div(id='graph_container'),
])

@callback(
    Output('graph_container', 'children'),
    [Input('submit_btn', 'n_clicks'),
     Input('Interval', 'n_intervals'),
      State('src_IP_options', 'value'),
     State('dst_IP_options', 'value'),
     State('prt_options', 'value'),
])
def Select_options(n_clicks, intervals, value1, value2, value3):
        
    files = glob.glob("C:/DataCapture/*.txt")
    modified_files = list()
    current_time = time.time()
    for txt_file in files:
        time_delta = current_time - os.path.getmtime(txt_file)
        if time_delta < 90:
            modified_files.append(txt_file)
    dfs = []
    for file_name in modified_files:
        df1 = pd.read_csv(file_name, on_bad_lines='skip').dropna()
        df1.rename(
        columns={'_ws.col.Time': 'Time', 'ip.src': 'Source', 'ip.dst': 'Destination', 'ip.proto': 'Protocol'},
                inplace=True)
        dfs.append(df1)
    df = pd.concat(dfs)
    list_src_IP = list(df['Source'].unique())
    list_dst_IP = list(df['Destination'].unique())
    list_prt = list(df['Protocol'].unique())
2

Your code is a bit hard to follow, but you can set the dropdown options via a callback, probably within the same callback you are already doing. Just use: Output(‘dst_IP_options’,‘options’).

3

Sorry, I understand it is not the best sample, but the code is a bit long, and as you said, hard to follow. Going back to your suggestion.

In the callback, I have the src_IP_options, dst_IP_options, and prt_options as input to callback, they are created outside the callback at the beginning, so if I understand you well, I should also have this in the output as well, so those three options will be passed back to the layout, is this correct? Or at least this is how I understand the callback. So in this case, the callback code will be something like this:

Output('graph_container', 'children'),
  Output('list_src_IP', 'value'),
 Output('list_dst_IP', 'value'),
 Output('list_prt', 'value'),
[Input('submit_btn', 'n_clicks'),
 Input('Interval', 'n_intervals'),
  State('src_IP_options', 'value'),
 State('dst_IP_options', 'value'),
 State('prt_options', 'value'),

is this correct? and if yes, then wouldn’t be a conflict between those variable? they were initially created outside the callback, and now sent back from the callback
If no, then my thinking of the process is wrong

Thanks again

4

You are going to want to use “options” in the output, not “value”. The dropdown has 2 (probably more) primary components:
the options (the dropdown list) and the value (the currently selected value out of the list). So I believe this what you’d want:

Output('graph_container', 'children'),
  Output('list_src_IP', 'options'),
 Output('list_dst_IP', 'options'),
 Output('list_prt', 'options'),
[Input('submit_btn', 'n_clicks'),
 Input('Interval', 'n_intervals'),
  State('src_IP_options', 'value'),
 State('dst_IP_options', 'value'),
 State('prt_options', 'value'),

The the variable you are altering is the options parameter of your dcc.Dropdown. While the initial value was created outside of the callback, the Dropdown was created within the callback. You should be in good shape!

5

Ok, it did not work, so I minimized my code to have a better picture, although it reads external files that changes every 10 seconds, so in reality you will not be able to run it locally
But the below code work in this way:
1- It creates a list of available ports starting from code line 10, saves it to list_prt in line 25
2- The layout uses the list to create the dropdown list, as in line 34
3- The data is processed inside the callback, line 55
4- A new port list is created based on the new processed data, line 74

I tested this code with this output:
1- The main capture is correctly picking up the existing ports at the start of the program and it renders the dropdown correctly
2- The second capture correctly picking up the new ports list and get updated every 10 seconds, I tested it using print, however the options in the dropdown list remains unchanged and based on the list created at the beginning of the program

Any idea?

Here is the code:

import dash
from dash import dcc, html, Input, Output, State, callback
import pandas as pd
import dash_bootstrap_components as dbc
import os, glob, time

app = dash.Dash(__name__, suppress_callback_exceptions=True, title='Network Monitoring Dashboard', external_stylesheets=[dbc.themes.BOOTSTRAP])
dash.register_page(__name__)
dash.Dash.suppress_callback_exceptions=True
# Initial Protocol list capture
path = r'C:/DataCapture/'
files_path = os.path.join(path, '*')
files = sorted(
    glob.iglob(files_path), key=os.path.getctime, reverse=False)
latest = files[-2: -8: -1]
df_IP = pd.DataFrame(columns=['Time', 'Source', 'Destination', 'Protocol'])
for file_name in latest:
    df1 = pd.read_csv(file_name, on_bad_lines='skip').dropna()
    df1.rename(
        columns={'_ws.col.Time': 'Time', 'ip.src': 'Source', 'ip.dst': 'Destination', 'ip.proto': 'Protocol'},
        inplace=True)
    df_IP = pd.concat([df_IP, df1], ignore_index=True)
    df_IP['Protocol'] = df_IP['Protocol'].replace({1: 'ICMP', 2: 'IGMP', 6: 'TCP', 17: 'UDP', 80: 'Http', 443: 'Https'})
    list_src_IP = list(df_IP['Source'].unique())
    list_prt = list(df_IP['Protocol'].unique())

app.layout = html.Div([
    html.H1('You can enter your filtering options below', className='text-center fs-1 text-primary'),
    dcc.Interval(
        id = 'Interval',
        interval = 5 *2000,
        n_intervals = 0
        ),
    dcc.Dropdown(
        id='prt_options',
        placeholder = 'Select The Port or leave empty',
        options = list_prt,
       style={"width": "60%"}
        ),
    html.Br(),
    html.Label(['']),
    html.Button(id='submit_btn', n_clicks=0, children='Submit'),
    html.Div(id='graph_container'),
])

@callback(
    [Output('prt_options','options')],
    Output('graph_container', 'children'),
    [Input('submit_btn', 'n_clicks'),
     Input('Interval', 'n_intervals'),
     State('prt_options', 'value'),
])
def Select_options(n_clicks, intervals, value1):
    # Data processing section
    files = glob.glob("C:/DataCapture/*.txt")
    modified_files = list()
    current_time = time.time()
    for txt_file in files:
        time_delta = current_time - os.path.getmtime(txt_file)
        if time_delta < 90:
            modified_files.append(txt_file)
    dfs = []
    for file_name in modified_files:
        df1 = pd.read_csv(file_name, on_bad_lines='skip').dropna()
        df1.rename(
        columns={'_ws.col.Time': 'Time', 'ip.src': 'Source', 'ip.dst': 'Destination', 'ip.proto': 'Protocol'},
                inplace=True)
        dfs.append(df1)
    df = pd.concat(dfs)
    df['Protocol'] = df['Protocol'].replace({1: 'ICMP', 2: 'IGMP', 6: 'TCP', 17: 'UDP', 80: 'Http', 443: 'Https'})
    df['Source'] = df['Source'].astype("string")
    df['Protocol'] = df['Protocol'].astype("string")
    # Reoccuring protocol list capture
    list_prt = list(df['Protocol'].unique())

if __name__ == "__main__":
    if not os.path.exists(r'c:/DataCapture'):
        os.mkdir(r'c:\DataCapture')
    app.run_server(debug=True, port = 8020)

Thanks

6

You aren’t returning anything in your def Select_options() .

You should be returning two things (in this order):

your dropdown options (which I think is list_prt) and the graph_container children, which I don’t see being generated.

edit: you weren’t returning anything in your first code, I just didn’t catch that. You have to be returning something in the callback that maps to your callback outputs.

7

Ok, I keep learning :slight_smile:

I added this to the end of the function: return list_prt , don’t worry about the graph for now

Now, for the return, which is the list_prt it still not updating the dropdown menu, even though I can see the list printing (with updates) on the console, but the menu is not updating

is the rest of the code (output and layout) correct?

8

Keep asking questions!

Did you remove Output(‘graph_container’, ‘children’) from you code? Because it will be looking for two outputs in the return.

Are you getting any errors?

9

I didn’t remove before and there was no errors, however, removing it did not make a difference

10

Hmm… can you post the updated code as well as what is printing out in the console? And when you say no errors… are you just looking at the console? You should have gotten an error when you are returning only one value to a two output callback.

The error should be in the UI (in the little dash toolbar in the bottom right).

11

Ok, here is the updated code, two changes, removed the Output of the graph container (line 48) and added the return (line76):

import dash
from dash import dcc, html, Input, Output, State, callback
import pandas as pd
import dash_bootstrap_components as dbc
import os, glob, time

app = dash.Dash(__name__, suppress_callback_exceptions=True, title='Network Monitoring Dashboard', external_stylesheets=[dbc.themes.BOOTSTRAP])
dash.register_page(__name__)
dash.Dash.suppress_callback_exceptions=True
# Initial Protocol list capture
path = r'C:/DataCapture/'
files_path = os.path.join(path, '*')
files = sorted(
    glob.iglob(files_path), key=os.path.getctime, reverse=False)
latest = files[-2: -8: -1]
df_IP = pd.DataFrame(columns=['Time', 'Source', 'Destination', 'Protocol'])
for file_name in latest:
    df1 = pd.read_csv(file_name, on_bad_lines='skip').dropna()
    df1.rename(
        columns={'_ws.col.Time': 'Time', 'ip.src': 'Source', 'ip.dst': 'Destination', 'ip.proto': 'Protocol'},
        inplace=True)
    df_IP = pd.concat([df_IP, df1], ignore_index=True)
    df_IP['Protocol'] = df_IP['Protocol'].replace({1: 'ICMP', 2: 'IGMP', 6: 'TCP', 17: 'UDP', 80: 'Http', 443: 'Https'})
    list_src_IP = list(df_IP['Source'].unique())
    list_prt = list(df_IP['Protocol'].unique())

app.layout = html.Div([
    html.H1('You can enter your filtering options below', className='text-center fs-1 text-primary'),
    dcc.Interval(
        id = 'Interval',
        interval = 5 *2000,
        n_intervals = 0
        ),
    dcc.Dropdown(
        id='prt_options',
        placeholder = 'Select The Port or leave empty',
        options = list_prt,
       style={"width": "60%"}
        ),
    html.Br(),
    html.Label(['']),
    html.Button(id='submit_btn', n_clicks=0, children='Submit'),
    html.Div(id='graph_container'),
])

@callback(
    [Output('prt_options','options')],
#    Output('graph_container', 'children'),
    [Input('submit_btn', 'n_clicks'),
     Input('Interval', 'n_intervals'),
     State('prt_options', 'value'),
])
def Select_options(n_clicks, intervals, value1):
    # Data processing section
    files = glob.glob("C:/DataCapture/*.txt")
    modified_files = list()
    current_time = time.time()
    for txt_file in files:
        time_delta = current_time - os.path.getmtime(txt_file)
        if time_delta < 90:
            modified_files.append(txt_file)
    dfs = []
    for file_name in modified_files:
        df1 = pd.read_csv(file_name, on_bad_lines='skip').dropna()
        df1.rename(
        columns={'_ws.col.Time': 'Time', 'ip.src': 'Source', 'ip.dst': 'Destination', 'ip.proto': 'Protocol'},
                inplace=True)
        dfs.append(df1)
    df = pd.concat(dfs)
    df['Protocol'] = df['Protocol'].replace({1: 'ICMP', 2: 'IGMP', 6: 'TCP', 17: 'UDP', 80: 'Http', 443: 'Https'})
    df['Source'] = df['Source'].astype("string")
    df['Protocol'] = df['Protocol'].astype("string")
    # Reoccuring protocol list capture
    list_prt = list(df['Protocol'].unique())
    print(list_prt)
    return list_prt

if __name__ == "__main__":
    if not os.path.exists(r'c:/DataCapture'):
        os.mkdir(r'c:\DataCapture')
    app.run_server(debug=True, port = 8020)

Regarding the errors, I am getting new one now, which I have not seen before (also a snapshot)
Picture3
:

Traceback (most recent call last):
  File "C:\Users\Ali\AppData\Local\Programs\Python\Python311\Lib\site-packages\flask\app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ali\AppData\Local\Programs\Python\Python311\Lib\site-packages\flask\app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ali\AppData\Local\Programs\Python\Python311\Lib\site-packages\dash\dash.py", line 1283, in dispatch
    ctx.run(
  File "C:\Users\Ali\AppData\Local\Programs\Python\Python311\Lib\site-packages\dash\_callback.py", line 465, in add_context
    flat_output_values = flatten_grouping(output_value, output)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ali\AppData\Local\Programs\Python\Python311\Lib\site-packages\dash\_grouping.py", line 35, in flatten_grouping
    validate_grouping(grouping, schema)
  File "C:\Users\Ali\AppData\Local\Programs\Python\Python311\Lib\site-packages\dash\_grouping.py", line 211, in validate_grouping
    SchemaLengthValidationError.check(grouping, full_schema, path, len(schema))
  File "C:\Users\Ali\AppData\Local\Programs\Python\Python311\Lib\site-packages\dash\_grouping.py", line 180, in check
    raise SchemaLengthValidationError(value, full_schema, path, expected_len)
dash._grouping.SchemaLengthValidationError: Schema: [<Output `prt_options.options`>]
Path: ()
Expected length: 1
Received value of length 3:
    ['TCP', 'UDP', 'ICMP']

and for the output, you can see in the console image that the list_prt is changing, while the dropdown menu is not:
Picture1
Picture2

12

Yea, take the brackets off your output. That change was made a while back, I didn’t think it would matter for a single Output, but it does. It’s expecting a single list [output1] where output1 is your prt_options list. Just tested it locally and that was the issue.

I would just remove all the brackets in the callback, you don’t need them anymore.

@callback(
    Output('prt_options','options'),
#    Output('graph_container', 'children'),
    [Input('submit_btn', 'n_clicks'),
     Input('Interval', 'n_intervals'),
     State('prt_options', 'value'),
])
1 Like
13

Thanks for the help, I really appreciate it.

I ended up separating the data processing and plotting into two callbacks, easier to manage.