Dynamic Use of Global Data

[russellthehippo]

You’re right, this will not work with multiple users or with multiple users.

Maybe follow some flowchart like this:

• If the data is updated frequently:
  • if the data is not very large:
    • —> just pull the data every time from SQL
  • if the data is large:
    • this is the hardest one, frequently updated large data: you can try using brain-plasma like mentioned below, or saving to disk, on some interval and then pulling values from disk each time instead of from SQL. But it’ll just be slow unless only a small part of the data are update. Then pull only updated parts of the data and append those pieces to something on disk or in brain-plasma
• if the data is not updated frequently:
  • if the data is not very large:
    • store the data on disk as a Parquet file and read it at each call
  • if the data is large (1,000,000 rows +):
    • use what I built for this exact problem: https://github.com/russellromney/brain-plasma is a shared memory object store to hold variables as “global” but giving all threads access (only Linux/Mac)
    • otherwise, figure out something else like from https://dash.plot.ly/sharing-data-between-callbacks examples e+ or Apache Arrow and Dash - Community Thread

Unless the data is very large, and especially if the server hosts the app and the database, pulling from SQL should be fine for every call. Just write a function and call it each time you need data, and cache the values (just for that user session).