# Gunicorn \>10x slower

**URL:** <https://community.plotly.com/t/gunicorn-10x-slower/74122>\
**Category:** Dash Python\
**Created:** [April 4, 2023, 1:15pm UTC](https://community.plotly.com/t/gunicorn-10x-slower/74122 "2023-04-04T13:15:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex.dumitriu](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/alex.dumitriu/32/23522_2.png) [@alex.dumitriu](https://community.plotly.com/u/alex.dumitriu)\
**Post date:** [April 4, 2023, 1:15pm UTC](https://community.plotly.com/t/gunicorn-10x-slower/74122/1 "2023-04-04T13:15:05Z")

</div>

I just finished going through the docs about using gunicorn, redis, celery and flower. I’ve managed to set everything up where I can see the tasks being queued in Flower. Cool. Dev environment is an M2 with 8 cores. `uvloop` is installed. `orjson` is installed. Python versions I tested with: 3.10.0, 3.11.2. Dash version is the latest one.

I have an app that runs locally, a portfolio optimizer that runs some heavy computations and returns a `JSON` to a `dcc.Store`. Some plotting callbacks then pull data from there and graph stuff out. Using `tqdm` on the loops inside the app, I see that for scenario X, I have 50 iterations/second. I’m using `polars` dataframes so I can see all cores are at 100%.

Adding `server=app.server` in `app.py` and running the server with `gunicorn -w 2 app:server` I can see the performance drop to a maximum of 2 iterations/second. CPUs are mostly idle. Changing the number of workers to 2\*cores + 1 (17) yields the exact same results. I guess it’s important to mention here that I don’t return anything until the end of the computation.

To test this differently I created a main.py that spawns a FastAPI() server. I have a GET method there that just replicates some compute logic from the Dash app. I call that method from my dash app with a `background_callback` since it takes longer than 30s.

Using `uvicorn main:app --log-level info --workers 17 --port 8001` for FastAPI(), the iterations are the same as before i.e. ~50 and everything runs error-free, CPUs at full blast.

Using the `gunicorn` production server `gunicorn -w 17 -k uvicorn.workers.UvicornWorker -b '127.0.0.1:8001' main:app` as suggested in their production-ready documentation (using gunicorn with Uvicorn workers), this again slows down to a max of 2 iterations per second, not to mention I get timeouts and errors like `resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown`.

I’ve tried all sorts of combinations from this forum and SO with workers types (including gevent), threads, and the results are the same. I’m sure I’m missing something here.

I understand from docs and forums that the dev servers are not configured to have workers and they just use everything that’s available but I don’t see a performance increase/drop by fiddling with the -w command in gunicorn.

---

<div class="post-metadata">

**Author:** ![jinnyzor](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/jinnyzor/32/21194_2.png) [@jinnyzor](https://community.plotly.com/u/jinnyzor)\
**Post date:** [April 4, 2023, 2:12pm UTC](https://community.plotly.com/t/gunicorn-10x-slower/74122/2 "2023-04-04T14:12:52Z")

</div>

Hello @alex.dumitriu,

Thank you for the details. 🙂 And welcome to the community!

Do you also have nginx working in front of the gunicorn?

I think the difference may be that FastAPI operates async while Flask operates sync, the workers should be picking up the difference, but I am wondering if it just is getting tied to your instance. 50 iterations is pretty quick, have you thought about possibly using websockets instead of performing post requests?

---

<div class="post-metadata">

**Author:** ![alex.dumitriu](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/alex.dumitriu/32/23522_2.png) [@alex.dumitriu](https://community.plotly.com/u/alex.dumitriu)\
**Post date:** [April 4, 2023, 3:36pm UTC](https://community.plotly.com/t/gunicorn-10x-slower/74122/3 "2023-04-04T15:36:01Z")

</div>

Hi there. Thanks for the reply.

No nginx. FastAPI was merely another test to see if performance dropped when using gunicorn with different colors (uvicorn workers). Ideally I would like to avoid FastAPI altogether.

The iterations I’m referring to are not requests but rather portfolio scenarios (just math operations visible through `tqdm`) and depending on the number of stocks in a portfolio they range from a few iterations to hundreds. I just set the portfolio to a fixed number of assets and started testing.

At this point I’ve reduced everything to a `background_callback` that takes a button as an `Input`, sends a request to a FastAPI function (no `async` in the signature, no need to) and runs some CPU bound logic. After a few minutes I get a JSON back from the request and store it in `dcc.Store`. Before going the API route this logic was just another plain old function in another module within the dash project. As soon as I start these 2 with `gunicorn` I can feel my beard growing. 🥲

I’m at the point where I would just like to see similar results between using the development server and gunicorn on my local machine. Same code, sync, nothing fancy.

Thanks for the support!

---

<div class="post-metadata">

**Author:** ![jinnyzor](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/jinnyzor/32/21194_2.png) [@jinnyzor](https://community.plotly.com/u/jinnyzor)\
**Post date:** [April 4, 2023, 3:39pm UTC](https://community.plotly.com/t/gunicorn-10x-slower/74122/4 "2023-04-04T15:39:21Z")

</div>

Do you have a basic MRE:

> [@How to write a Minimal Reproducible Example (MRE)](https://community.plotly.com/t/how-to-write-a-minimal-reproducible-example-mre/61502):
>
> Adding a minimal reproducible example to your questions will help attract community members to answer your question. The goal of writing a MRE is to allow other members of the community to run the code on their computers and “reproduce” the error, so that they can best help you. Please make sure that your MRE includes data and that the code is correctly pre-formatted. To preformat, copy and paste the code in your post and then click on \</\> (Preformatted text), as shown on the image:

This is very interesting. 🙂

You can replace the FastAPI calls with using a websocket, put out by dash-extensions.

---

<div class="post-metadata">

**Author:** ![alex.dumitriu](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/alex.dumitriu/32/23522_2.png) [@alex.dumitriu](https://community.plotly.com/u/alex.dumitriu)\
**Post date:** [April 5, 2023, 6:23am UTC](https://community.plotly.com/t/gunicorn-10x-slower/74122/5 "2023-04-05T06:23:33Z")

</div>

Ok so here’s an example I put together this morning:

app.py:

```auto
import os
import secrets

import diskcache
import ujson as json
from celery import Celery
from dash import Dash
from dash import DiskcacheManager, CeleryManager, Input, Output, html, dcc

from computations.computations import compute

secret = secrets.token_urlsafe(32)

if 'REDIS_URL' in os.environ:
    # Use Redis & Celery if REDIS_URL set as an env variable
    celery_app = Celery( __name__ ,
                        broker='redis://127.0.0.1:6379/0',
                        backend='redis://127.0.0.1:6379/1')

    background_callback_manager = CeleryManager(celery_app)

else:
    # Diskcache for non-production apps when developing locally
    cache = diskcache.Cache("./cache")
    background_callback_manager = DiskcacheManager(cache)

app = Dash( __name__ , background_callback_manager=background_callback_manager)

app.layout = html.Div(
    [
        dcc.Store(id="intermediate-value", storage_type="session"),
        html.Button('Submit', id='submit-button'),
        html.Button('Cancel', id='cancel-button'),
    ]
)

app.secret_key = secret
server = app.server

@app.callback(
    output=Output("intermediate-value", "data"),
    inputs=[Input("submit-button", "n_clicks")],
    background=True,
    prevent_initial_call=True,
    running=[(Output("submit-button", "disabled"), True, False)],
    cancel=Input("cancel-button", "n_clicks"),
)
def gen_results(n_clicks: int) -> json:
    print(f"n_clicks: {n_clicks}")
    result = compute()
    print(f"result: {result}")
    return json.dumps(result)

if __name__ == " __main__":
    app.run(port=8050, debug=True)

```

computations.py - dummy code to simulate work:

```auto
from tqdm import tqdm
import random

def compute() -> dict:
    L = range(1_000_000)
    amount = 1_000_000
    random_array = [random.choice(L) for _ in range(amount)]
    
    for _ in tqdm(random_array):
        file = open("testfile.txt", "w")
        file.write(f"Hello World{str(_)}")
        file.close()

    return {"job": "finished"}

```

Start server with: `gunicorn app:server`  
Start celery with: `celery -A app.celery_app worker --loglevel=INFO`

In this example it seems I am getting the same speeds for both werkzeug and gunicorn. Weirdly enough I’m seeing the `tqdm` progress bar inside celery and not the server terminal. I think I need a break 🫠. I don’t understand how only changing one line of code from `python app.py` to `gunicorn app:server` can cut so much power.

One question for this example though: Should I see performance differences when fiddling with the number of workers in gunicorn? Right now regardless if I put in 1 or 17 workers the number of iterations/sec I get from `compute()` is the same.

---

<div class="post-metadata">

**Author:** ![alex.dumitriu](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/alex.dumitriu/32/23522_2.png) [@alex.dumitriu](https://community.plotly.com/u/alex.dumitriu)\
**Post date:** [April 5, 2023, 1:55pm UTC](https://community.plotly.com/t/gunicorn-10x-slower/74122/6 "2023-04-05T13:55:11Z")

</div>

Quick update. I recently changed to `polars` from `pandas`. The gains in speeds on the DF operations themselves in my case have been significant (\>10x), but loading up an old branch that used pandas, it seems everything runs fine in terms of server behaviour. Identical speeds for `werkzeug` and `gunicorn`. Nothing changes by tuning the number of workers though.

The reduced performance when switching to gunicorn seems to be happening only with `polars` frames. Chopping up the code I see there’s also some weird behavior with `background_callbacks` and `polars`.

This hangs without any errors.

```auto
@app.callback(
    Output("intermediate-value", "data"),
    Input("submit-button", "n_clicks"),
    background=True,
    prevent_initial_call=True,
    running=[(Output("submit-button", "disabled"), True, False)],
    cancel=Input("cancel-button", "n_clicks"),
)
def test(n_clicks: int):
    data = pl.read_parquet("file.parquet")
    return data.write_json()

```

This doesn’t. I’ve added `parallel='none'` to `read_parquet()`:

```auto
@app.callback(
    Output("intermediate-value", "data"),
    Input("submit-button", "n_clicks"),
    background=True,
    prevent_initial_call=True,
    running=[(Output("submit-button", "disabled"), True, False)],
    cancel=Input("cancel-button", "n_clicks"),
)
def test(n_clicks: int):
    data = pl.read_parquet("file.parquet", parallel='none')
    return data.write_json()

```

Making this a regular callback however works with the default `parallel='auto'`:

```auto
@app.callback(
    output=Output("intermediate-value", "data"),
    input=Input("submit-button", "n_clicks"),
)
def test(n_clicks: int):
    data = pl.read_parquet("file.parquet")
    return data.write_json()

```

After reading the files with `parallel='none'`, I join the DFs using `.join` and once again in a background callback it chokes without errors. The task in `celery` is active but the processor is idle. I have to cancel the task.

Something to be mindful of. I’m not sure how to get around this. Since FastAPI also uses `gunicorn` I’m not sure how I can continue using `polars`. Any suggestions? How are you guys using polars in dash?

---

<div class="post-metadata">

**Author:** ![jinnyzor](https://sea2.discourse-cdn.com/flex024/user_avatar/community.plotly.com/jinnyzor/32/21194_2.png) [@jinnyzor](https://community.plotly.com/u/jinnyzor)\
**Post date:** [April 5, 2023, 2:12pm UTC](https://community.plotly.com/t/gunicorn-10x-slower/74122/7 "2023-04-05T14:12:44Z")

</div>

I actually don’t use polars, because I handle a bunch of sql queries. When last I looked, it seemed that reading from a sql query required the variables to be inserted into the string.

This is a big no-no to avoid when taking things that can be user defined. Because it can lead to sql injection attacks.

FastAPI can also use uvicorn.