Error: The quota has been exceeded

So I have an app, where I upload a csv file, i can make comments to it, and it stores it in a different division to retrieve for some other processes.
My app runs smoothly, until I submit the first comment and a copy of the dataframe will be stored.
After that the App is extremely slow, for processes which do not even directly use this dataframe, so they should not be slower.
So I thought, maybe there is just too much stuff stored in the App.
I changed from hidden html.divs for data storage to the dcc.Store component and stored the dataframe locally.
This did not change anything.
I noticed that, I could reduce the amount of stored copies from 3 to 2, but this did not change anything.
Then I noticed that in the browser, but not in the terminal I get the Error: The quota has been exceeded, With the below specifications.
What does this mean, and how can I potentially fix it?

(This error originated from the built-in JavaScript code that runs Dash apps. Click to see the full stack trace or open your browser's console.)

value@http://127.0.0.1:8050/_dash-component-suites/dash_core_components/dash_core_components.min.js?v=1.0.0&m=1561723217:47:429623

value@http://127.0.0.1:8050/_dash-component-suites/dash_core_components/dash_core_components.min.js?v=1.0.0&m=1561723217:47:431684

qi@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react-dom@16.8.6.min.js?v=1.0.0&m=1561723216:130:296

ui@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react-dom@16.8.6.min.js?v=1.0.0&m=1561723216:133:320

Vc/<@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react-dom@16.8.6.min.js?v=1.0.0&m=1561723216:158:377

unstable_runWithPriority@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react@16.8.6.min.js?v=1.0.0&m=1561723216:27:36

Vc@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react-dom@16.8.6.min.js?v=1.0.0&m=1561723216:158:360

Sc@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react-dom@16.8.6.min.js?v=1.0.0&m=1561723216:158:144

Z@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react-dom@16.8.6.min.js?v=1.0.0&m=1561723216:156:492

Kc@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react-dom@16.8.6.min.js?v=1.0.0&m=1561723216:155:69

ya@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react-dom@16.8.6.min.js?v=1.0.0&m=1561723216:153:185

enqueueSetState@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react-dom@16.8.6.min.js?v=1.0.0&m=1561723216:202:409

t.prototype.setState@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/react@16.8.6.min.js?v=1.0.0&m=1561723216:20:433

handleChange@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/dash_renderer.dev.js?v=1.0.0&m=1561723216:32559:14

dispatch@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/dash_renderer.dev.js?v=1.0.0&m=1561723216:33559:7

createThunkMiddleware/</</<@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/dash_renderer.dev.js?v=1.0.0&m=1561723216:33047:16

dispatch@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/dash_renderer.dev.js?v=1.0.0&m=1561723216:33105:18

handleResponse@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/dash_renderer.dev.js?v=1.0.0&m=1561723216:37587:17

handleJson@http://127.0.0.1:8050/_dash-component-suites/dash_renderer/dash_renderer.dev.js?v=1.0.0&m=1561723216:37702:47

Okay so I tried the same thing in Chrome instead of firefox and the error is a bit more informative here:

Failed to execute 'setItem' on 'Storage': Setting the value of 'flag_storage' exceeded the quota.

Googling that error message I found: https://stackoverflow.com/questions/23977690/local-storage-in-chrome
So it does seem, like this is a storage error, which is weird, since, as I said I also reduced the data storage by a third, and I did not have such problem with the same amount of data stored before.

So how can I solve this? It seems like caching might be an option https://dash.plot.ly/sharing-data-between-callbacks, but I do not really understand how to do it.
The examples are very convoluted and I do not understand how caching actually gets its inputs, how it is triggered or any of that stuff…

I guess it would make more sense to cache the original upload, as it is less used, then the flagged data.
How can i cache that callback, or more preferentially only the

dcc.Store(id='df_storage', storage_type='session', 
                  data=df.to_json(date_format='iso', orient='split'))

part?

#%% upload function
def parse_contents(contents, filename, date):
    '''
    parses contents of spreadsheath
    '''

    content_type, content_string = contents.split(',')

    decoded = base64.b64decode(content_string)
    try:
        
        if 'csv' in filename:
            # Assume that the user uploaded a CSV file
            df = pd.read_csv(
                io.StringIO(decoded.decode('utf-8')))
        elif 'xls' in filename:
            # Assume that the user uploaded an excel file
            df = pd.read_excel(io.BytesIO(decoded))
            
    except Exception as e:
        print(e)
        return html.Div([
            'There was an error processing this file.'
        ])
    #selection of graphs
    return html.Div([
        html.H5(filename),
        html.H6(datetime.datetime.fromtimestamp(date)),

        MD.generate_table(df[0:20]),
        #horizontal line
        html.Hr() ,
        dcc.Store(id='df_storage', storage_type='session', 
                  data=df.to_json(date_format='iso', orient='split'))])
    


#%% update after data upload
         
@app.callback(Output('output-data-upload', 'children'),              
              [Input('upload-data', 'contents')],
              [State('upload-data', 'filename'),
               State('upload-data', 'last_modified')])
#calls the upload function for storing  the data

def update_output(list_of_contents, list_of_names, list_of_dates):
    print(list_of_names)
    print(list_of_dates)
    if list_of_contents is not None:
        children = [
            parse_contents(c, n, d) for c, n, d in
            zip(list_of_contents, list_of_names, list_of_dates)]
        return children

One more thing: Before I had just saved the original dataframe in a global variable, because in the beginning I did not understand the data storage very well.
So interestingly this has never caused problems, the app to run slow any anything therelike.
But now, when I want to store it in a storage variable it gives the same error, but I can easily revert back and stored as a global variable it does not cause an error.

This whole thing really completely breaks my app, so I would very much appreciate any solution to this.

Do you need the storage to be persistent across page reloads? If not, storage_type='memory' would avoid the quota problem (which is browser-dependent but normally 2-10MB)

But that wouldn’t address the slowness issue, unless the slowness comes from trying to interpret a data file that got cut off in the middle.

Looks like we don’t have any examples in Part 4. Sharing Data Between Callbacks | Dash for Python Documentation | Plotly of caching uploaded files - We should definitely add that! The general pattern I’d use for that is:

One more thing: Before I had just saved the original dataframe in a global variable, because in the beginning I did not understand the data storage very well.
So interestingly this has never caused problems, the app to run slow any anything therelike.

This will work fine as long as you’re the only one using the app, and you run the app in a single process. Once you have multiple users or processes though you’ll encounter problems.

Thank you for your answer. I will try with the caching later.
For now I used an ugly hack to just save the created file in the working directory and reload it when needed.
Clearly I have no idea how the browser uses local storage at all.
How does it make a difference if I set something as a global variable or store it in a division?
I had already tried to set all the dcc.Store storage types to ‘memory’ but that had not changed anything.

Talking about this: I do make graphs from my data, holding a lot of data points, which of course are displayed
in a division. As my input csv file already is ~12mb big I could imagine the graphs being of similar size, which could also cause the problem, although I never encountered it before creating that copy of the dataframe.
Also Chrome seems to perform better, not having the issue anymore, after storing the dataframe on the disk, while firefox still gets super slow. This is not exclusive to the window running the app, the whole browser gets super slow and unresponsive after some time.
It is really weird, because none of the processes are dependent on the copy of the dataframe.
All functions do the same with and without the copy of the dataframe. Just the source they are getting their data from changes.