Hi all,
I am having trouble with the backend architecture of my Dash app.
My desired functionality is:
- Every two weeks, an ML hyperparameter validation task is run. These parameters are supplied to the front end, and cached for two weeks, until the next time this task is run.
- Data is from AWS DynamoDB every the beginning of every hour, and cached until the next time this task is run.
Currently, I have a dcc.Interval component that run every hour. The results are cached using flask_caching with Redis as the backend. Currently, I have no control over when in the hour dcc.Interval fires. I have it setup nearly identical to the example Part 4. Sharing Data Between Callbacks | Dash for Python Documentation | Plotly.
Additionally, I want to employ a job queue with Celery, while maintaining the cache functionality. Say Person 1 uses my app and they run the expensive ML hyperparameter validation task and the results are returned to the queue. I want Person 2 to NOT run the ML task and instead read directly from the queue.
The ML tasks is very expensive (takes about 5 minutes) so I would like to set this up in the background, asynchronously, and in another thread.
How do I go about doing these things?