Thanks for the very detailed response! Yes this totally make sense.
I am already using gunicorn with the solution you have proposed with multiple workers and I am very happy with it.
I think using celery has two advantages over the current solution:
I noticed that with the current logic the client when updating the component(s) wait for a request to reply (the POST to _dash-update-component). This may be killed if your webserver (for example ngnix) doesn’t have a timeout set to a large value( proxy_read_timeout). Sometimes you cannot change that, for example if your webserver is behind another proxy that your institution is managing (common case in academia). My understanding is that during this time that worker cannot serve another request.
I didn’t know about the async workers, very cool concept! In this case I guess I may still have the timeout problem with of the hanging request to to _dash-update-component, although the server is free to use that worker to do something else.With celery instead you can schedule the long blocking process and return immediately. In this case we can use your new loading css and logic to disable the interface and then the dcc.Interval in a callback that query the celery queue for task completion; once it is done you can update the component that may show the results and re-enable the interface. In this case there is no hanging request on the client side (since you send a period request thanks to the interval) so we can use the default timeouts of ngnix and gunicorn.
If you have many users the advantage of using celery queue is that you can serve more users than the number of gunicorn workers you have since the workers are not constantly busy with long blocking processes. I like celery since you can specify the number of tasks that can be executed simultaneously, but you can still take-in more tasks and append them to the queue. So the server will not crash if your long blocking task are taking too much memory/cpu since you know for sure how many celery task will run at a given time. The idea is to have multiple gunicorn workers and only 1 or 2 celery worker. With celery you can also send the task to another machine so you can easily scale the slow part of the app if necessary. I used this solution with a previous webapp I developed a while ago with Celery +Flask+bootstrap (http://crispresso.rocks/)
For bioinformatics task there are a lot of apps already built with Shiny for gene expression analysis, single cell, genomics etc, some examples here:
Shiny is the currently the standard the facto for this kind of apps, but I personally like your project more, so I really appreciate your time to reply to all the users and keep the community so engaged. I am based in Boston (pinellolab.org), if you are visiting as some point we should take a coffee. Happy to tell you more about how we are using Dash for our research projects.
Thanks again for your help