NGINX-Uwsgi-Django deployment with multiprocess fails. With monoprocess works (Unable to find stateless DjangoApp)

I have a stack that looks like this:

NGINX (nginx.conf):

    worker_processes 8;
    events {
        worker_connections 1024;
        use epoll;
        multi_accept on;
        accept_mutex off;
    }
    upstream django {
        server 127.0.0.1:8001;
        server 127.0.0.1:8002;
        server 127.0.0.1:8003;
        server 127.0.0.1:8004;
        server 127.0.0.1:8005;
        server 127.0.0.1:8006;
        server 127.0.0.1:8007;
        server 127.0.0.1:8008;
    }

    location /api {
      uwsgi_pass django;
      uwsgi_read_timeout 600;

      uwsgi_param Host $host;
      uwsgi_param X-Real-IP $remote_addr;
      uwsgi_param X-Forwarded-For $proxy_add_x_forwarded_for;
      uwsgi_param X-Forwarded-Proto $http_x_forwarded_proto;

      uwsgi_param QUERY_STRING $query_string;
      uwsgi_param REQUEST_METHOD $request_method;
      uwsgi_param CONTENT_TYPE $content_type;
      uwsgi_param CONTENT_LENGTH $content_length;
      uwsgi_param REQUEST_URI $request_uri;
      uwsgi_param PATH_INFO $document_uri;
      uwsgi_param DOCUMENT_ROOT $document_root;
      uwsgi_param SERVER_PROTOCOL $server_protocol;
      uwsgi_param HTTPS $https if_not_empty;
      uwsgi_param REMOTE_ADDR $remote_addr;
      uwsgi_param REMOTE_PORT $remote_port;
      uwsgi_param SERVER_PORT $server_port;
      uwsgi_param SERVER_NAME $server_name;
   }

The uwsgi files are all identicall except for the corresponding sockets. They are like this (mysite.xml):

<uwsgi>
    <socket>127.0.0.1:8001/api</socket>
    <chdir>/var/www/MY_API</chdir>
    <module>nimgenetics.wsgi:application</module>
    <debug>true</debug>
</uwsgi>

When I run uwsgi with a single process, it works fine. But when I run uwsgi with several processes on the different sockets [8001-8008], the whole app works fine except the Dash graphs. I get the following error:

Unable to find stateless DjangoApp called app

Any guesses of how can I solve this in order to run Dash with multiprocessing with NGINX+UWSGI+Django?

Where are you importing your app?

In other words, which file has the import statement, as it sounds like it is not being imported all of the time.

Hi, @delsim. I have included the graph import on my urls.py file as follows:

from .plots import genome_array

(being genome_array my graphs file)

It works ok when running it monoprocess, the this is when going multiprocess :confused:

Can you share the file with your app in it? It does sound like the app is only created in a subset of your running processes.

Yes, the Dash graphs are part of a greater project. The endpoint used for the Dash app is as follows:

views.py

@api_view(['GET'])
def chart_genome(request):
    # Fill kwargs with the info from the request to tell which case should be plotted and some info such as the language 
    generate_plot(**kwargs)  
    return render(request, 'plots/genome.html')

genome.html

<html>
    <head>
         <meta charset="utf-8" /> 
    </head>
    <body>
        {% load plotly_dash %}
        {%plotly_app name="Genome" ratio=1 %}
    </body>
</html>

genome.py

def generate_plot(**kwargs):

    app = DjangoDash('Genome', prevent_initial_callbacks = False)

    # Get fig_genome with scatterplot + kwargs + pandas querying

    try:
        app.layout = html.Div(children=[
            dcc.Store(id='kwargs'),

            html.Div([
                    # GENOME
                    html.Div([
                        html.Div([
                            dcc.Graph(
                                id='genome-graph',
                                figure=fig_genome,
                            ),
                        ]),

                    ],
                    ),
                ]
            ),

        ], style={'text-align': 'center'})

    except Exception as err:
        print(err)

Also, in urls.py, I have included the neccesary Dash urls:
url('api/django_plotly_dash/', include('django_plotly_dash.urls'))

So your problem is that generate_plot is not always called, so in some processes there is no Genome dash instance. If you invoke it at the bottom of genome.py, so it is called on every import (and thus in every process when Django imports all of the urls), your multiprocess error should be resolved.

If you need different kwargs content when calling this function then you’ll need to use stateful, not stateful, Dash instances.

Hi, thanks a lot for the answer, it was really helpful.

The thing is that the frontend calls to this endpoint providing the parameter of the case that should be plotted, for instance:

User A: /api/genome?ID_CASE=1234
User B: /api/genome?ID_CASE=5678
  1. If I call generate_plot on every instance passing as kwargs param ID_CASE=1234, every one of them will have an instance of this graph, so the cases will get missed among the different users, no? User B will get the case of user A, right? If so, is there any way to fix this? Because the thing is that I do not know how to make the ID_CASE param also visible for the internal dash calls such as update_layout, which I do not know if it would be neccesary.

  2. On the other hand, I am wondering: I did this this way because I needed to pass the param ID_CASE from the Django view to the Dash graph. Is it possible to have a stateless app where you could pass them this input param that is unique for each user?

Thanks a lot.

So you’re saying that the app has internal state; for this you can use a stateful DashApp instance.

You will need to think about how you share/obtain the data; I’d go for an approach where the app contains the ID_CASE reference and then access the data associated with it in the callbacks, probably leveraging Django’s caching if getting this data is expensive.

1 Like

Thanks a lot for the reply, @delsim. After some tryouts, I managed to solve this problem.