Using multiprocessing.Pool in callbacks

Dash Python
1

Hello everyone,

I am wondering if anybody has managed to use multiprocessing inside of a callback in a Dash application, I am particularly trying to use Pool but have also tried with Process.
My application is running inside of a Docker container and what I have found is that whenever the callback gets to the code trying to do multiprocessing, it is left hanging forever.

I have put this simple example together to illustrate what I am trying to do:

from multiprocessing import Pool
import time
import dash
from dash import html, callback, Input, Output
import dash_bootstrap_components as dbc

page_static_name = "test_page"
dash.register_page(
    __name__,
    path_template="/v3/test_page",
    title="test",
    name="test",
    description="test page",
)


def get_element(name):
    print("starting with ", name)
    time.sleep(0.2)
    return html.P(name, className="border")


def layout():
    return dbc.Container(
        [
            dbc.Row(
                [dbc.Col([dbc.Button("Make output", n_clicks=0, id="make_output_btn")])]
            ),
            dbc.Row([dbc.Col([], id="output_div", width=2)]),
        ],
        fluid=True,
        className="mb-5",
    )


@callback(
    Output("output_div", "children"),
    Input("make_output_btn", "n_clicks"),
    prevent_initial_call=True,
)
def make_output(n_clicks):
    print("Callback started")
    elements = [f"element_{i}" for i in range(10)]
    output = []
    with Pool(5) as pool:
        print("Pool created")
        for element in elements:
            print(f"working on {element}")
            this_element = pool.apply_async(get_element, args=(element,))
            output.append(this_element)

    output = (element.get() for element in output)
    return output

In this example, the callback starts properly but then I find “Pool created” is never printed and the app is stuck working endlessly.

As mentioned before, Process has not been successful either and I have also tried changing the spawn method to all options listed in the multiprocessing documentation. If anybody has managed to implement something like this and running it in a container, I would greatly appreciate some guidance.

Thanks!

3

Hoping this is helpful - the short self-contained example below does appear to work, at least on the surface.

I’ll add though, if you replace my p = Pool() with with Pool() as p: then it does hang as you described, and I’ve no idea why that change should make any difference.

from multiprocessing import Pool
from dash import html, Dash, callback, Input, Output

app = Dash(__name__)
app.layout = html.Div(
    [
        html.Button("Calculate", id="button-calc"),
        html.Button("Clear", id="button-clear"),
        html.Div(id="div-output"),
    ]
)

def f(x):
    return x*x

@callback(
    Output("div-output", "children", allow_duplicate=True),
    Input("button-calc", "n_clicks"),
    prevent_initial_call=True
)
def calc(_):
    p = Pool()
    result = p.map(f, [1, 2, 3])
    p.terminate()
    return ",".join([str(i) for i in result])

@callback(
    Output("div-output", "children", allow_duplicate=True),
    Input("button-clear", "n_clicks"),
    prevent_initial_call=True
)
def clear(_):
    return ""

if __name__ == '__main__':
    app.run_server(debug=False,  host='0.0.0.0')