How to deal with huge states?

Hello,

Here is how my code works:

  • the user enters some parameters and starts a very expensive calculation
  • the result of this calculation is a very large array looking like:
    [{"step":1, ...}, ... {"step":50000, ...}]
  • one step can be displayed on screen at a time (there is a button to jump to step x)

Here is approximately how I do:

  • I put the result of the calculation inside Store(id=“result”).
  • I create a callback that works approximately like that:
@app.callback(Output("display", "children"), Output("result", "data"), [Input("jump-to-step-x-btn", "value")], [State("result", "data")])
def foo(step, result):
    if ...:
        result = ... # possibly update the array
    return result[step], result

My problem is that, since the array is very big (nearly 500000 characters), it lags a little bit (approx. 0.5 sec) when the user tries to jump to a different step. My guess is that it is because the array has to be sent and then retrieved from the server.

Any advice on what would be a good way to fix this issue?
Thank you very much and have a nice week.