Background call with pytorch crashes

Dash Python
1

Hi,

I am trying to implement a background call that leads to evaluation of a pytorch command:
data = torch.tensor(data, dtype=torch.float32).to(self.device)

The process spawned by the background call seems to crash without a hint of an error anywhere. The execution then returns to Dash without an indication of a problem.

Without the background call, the code executes without any issue. I’m using DiskcacheLongCallbackManager.

Any idea what might be the issue or how to debug it?

2

Hi Jack,

is your device a GPU or CPU ? Could imagine that using a GPU would break the process, maybe try to convert your result back to CPU if you’re using one - with torch.Tensor.cpu

I hope this helps

3

Good idea, thanks! I forgot to say that I tried this and it didn’t help unfortunately.
It’s not easy to debug because the process really seems to just crash, there’s no exception raised…

4

Hm okay too bad haha. Can you provide a minimum example of your backgroundcallback ?

5

I think I’ve been able to figure this out. It’s a bug in Python multiprocessing, resp. maybe in MacOS.

This is the minimal example:

from dash import Dash, html, dcc, callback, Output, Input
from dash.exceptions import PreventUpdate
import plotly.express as px
import torch
import time


from dash.long_callback import DiskcacheLongCallbackManager
import diskcache
cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)
background_callback_manager=long_callback_manager 


app = Dash(background_callback_manager=background_callback_manager)

app.layout = [
    html.H1(children='Pytorch in Background Test', style={'textAlign':'center'}),
    html.Button("To mps", id='to-mps-button'), 
    dcc.Loading(children=html.Div(id='output'), type="circle"),
    html.Hr(),
    html.Button("To cpu", id='to-cpu-button'), 
    dcc.Loading(children=html.Div(id='output-2'), type="circle"),
]

@callback(
    Output('output', 'children'),
    Input('to-mps-button', 'n_clicks'),
    background=True
)
def to_mps(n_clicks):
    if not n_clicks:
        raise PreventUpdate
    time.sleep(1)

    print("About to transfer to mps")
    data = [[1, 2, 3], [4, 5, 6]]
    data = torch.tensor(data, dtype=torch.float32).to("mps")
    print("To mps completed", n_clicks)
    return "To mps completed " + str(n_clicks)

@callback(
    Output('output-2', 'children'),
    Input('to-cpu-button', 'n_clicks'),
    background=True
)
def to_cpu(n_clicks):
    if not n_clicks:
        raise PreventUpdate
    time.sleep(1)
  
    print("About to transfer to CPU")
    data = [[1, 2, 3], [4, 5, 6]]
    data = torch.tensor(data, dtype=torch.float32).to("cpu")  
    print("To CPU completed", n_clicks)
    return "To CPU completed " + str(n_clicks)


if __name__ == '__main__':
    app.run(debug=True)

requirements.txt:

dash
dash-bootstrap-components
dash[diskcache]
psutil
diskcache
pandas

…and install pytorch via conda.

I was originally working with Python 3.9.19. After some poking around, I was able to get to the following error:

objc[38493]: +[NSCheapMutableString initialize] may have been in progress in another thread when fork() was called.
objc[38493]: +[NSCheapMutableString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

…which points to e.g.:

https://bugs.python.org/issue33725

However, the suggested workaround OBJC_DISABLE_INITIALIZE_FORK_SAFETY = YES doesn’t work with Python 3.9.19.

The problem goes away with Python 3.12.7 and with OBJC_DISABLE_INITIALIZE_FORK_SAFETY = YES set.

Just for completeness, this is what the test looks like:

image
image862×406 28.2 KB

…pressing To CPU twice, then To mps once
In the terminal:

Dash is running on http://127.0.0.1:8050/

 * Serving Flask app 'minimal'
 * Debug mode: on
About to transfer to CPU
To CPU completed 1
About to transfer to CPU
To CPU completed 2
About to transfer to mps
objc[80721]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[80721]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.