Storing multiprocessing.Process() and multiprocessing.Queue()

Dash Python
1

I am building a machine learning application with Dash Python. The main process is running the Dash app. From there, I can start the ML training via a button click. Clicking on the button starts a multiprocessing.Process() to run the ML training. I am using a multiprocessing.Queue() to send current training data to the Dash app.

I need to be able to terminate the training process and to close the queue when another button is clicked. In other words: I want to start and stop ML training as often as I want to, either starting a new multiprocessing.Process()and multiprocessing.Queue() or terminating/closing the currently running. Therefore, I am looking for a solution to store the multiprocessing.Process() and the multipprocessing.Queue() to be able to access them. However, this is not possible with dcc.Store since both object types are not JSON serializable.

Using global variables is not an option as it is bad practice and I have to create the multiprocessing.process dynamically with different parameters.

Is there any way to accomplish this?

2

Hello @frix,

I havent done this, but I dont think there is a way to avoid a global variable, because it has to have some way to reference it… Unless you can query a process by an id of sorts, in which case you can query it. :slight_smile:

But, if you cant, you can store it in a dictionary and pass the key to the dash application via a dcc.Store, just remember to delete the key after it is finished to make sure your memory stays clean.

3

Hi @jinnyzor ,

thanks for your reply! I found a very simple solution for the process related issue:
I use multiprocessing.Process.pidto get the process id and store it in dcc.Store.
In the callbeck where I need to kill the process, I use psutil.Process(pid) to get the process.

For the queue however, I haven’t found a solution so far. Currently, I am using a global variable. After the process has been killed, I empty the queue with a while loop so that it is empty when a new training process has been started:

while not q.empty(): q.get()

1 Like