multiprocessing Pool workers cannot create subprocesses
In a recent application, a student of mine tried to create child processes inside of a multiprocessing Pool worker (for security and convenience reasons, we wanted to run some code inside of a child process). Here is some test code for python 2.7: = import multiprocessing as mp def run_computation(x): print 2*x def f(x): curr_proc=mp.current_process() # uncomment following line to get this to work #curr_proc.daemon=False p = mp.Process(target=run_computation, args=(x,)) p.start() p.join() pool = mp.Pool(processes=4) pool.map(f, range(10)) === The result is: Traceback (most recent call last): File "daemon.py", line 17, in pool.map(f, range(10)) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 199, in map return self.map_async(func, iterable, chunksize).get() File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 491, in get raise self._value AssertionError: daemonic processes are not allowed to have children The problem appears to be that multiprocessing sets its workers to have the daemon flag set to True, which prevents workers from creating child processes. If I uncomment the line indicated in the code, I can create child processes and the program works (it prints out the even numbers from 0 to 18). It makes me nervous to just change the daemon status of the process like that, especially when I don't know the reason the workers have daemon=True to begin with. What is the reasoning behind that decision? What issues do we need to worry about if we just set the daemon mode flag like in the above code? Thanks, Jason -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing Pool workers cannot create subprocesses
On 3/18/11 3:29 PM, Ned Deily wrote: In article<4d838d28.5090...@creativetrax.com>, Jason Grout wrote: The problem appears to be that multiprocessing sets its workers to have the daemon flag set to True, which prevents workers from creating child processes. If I uncomment the line indicated in the code, I can create child processes and the program works (it prints out the even numbers from 0 to 18). It makes me nervous to just change the daemon status of the process like that, especially when I don't know the reason the workers have daemon=True to begin with. What is the reasoning behind that decision? What issues do we need to worry about if we just set the daemon mode flag like in the above code? http://docs.python.org/library/multiprocessing.html#multiprocessing.Proce ss.daemon "When a process exits, it attempts to terminate all of its daemonic child processes. Note that a daemonic process is not allowed to create child processes. Otherwise a daemonic process would leave its children orphaned if it gets terminated when its parent process exits. Additionally, these are not Unix daemons or services, they are normal processes that will be terminated (and not joined) if non-daemonic processes have exited." Right; thanks. Let me rephrase my questions: 1. Why is important that the multiprocessing Pool worker processors have daemon=True (I think this is the same as asking: why is it important that they be terminated with terminate() rather than join() )? 2. Is it safe for us to reset a Pool worker process to have daemon=False before starting a subprocess from that worker, like in the code from the original message? Thanks, Jason -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing Pool workers cannot create subprocesses
On 3/19/11 4:17 PM, John L. Stephens wrote: On 3/18/2011 7:54 PM, Jason Grout wrote: Right; thanks. Let me rephrase my questions: 1. Why is important that the multiprocessing Pool worker processors have daemon=True (I think this is the same as asking: why is it important that they be terminated with terminate() rather than join() )? 2. Is it safe for us to reset a Pool worker process to have daemon=False before starting a subprocess from that worker, like in the code from the original message? Thanks, Jason Jason, I just happen to be dealing with a project that uses multiprocessing. What I have learned is this... If a child thread (pool worker) is not set to daemon (daemon=False), if for some reason the parent thread terminates either normally or abnormally and the worker thread has not completed its task, the child thread will terminate by throwing all sorts of nasty errors. However, in daemon mode, multiprocessing will terminate the child thread 'cleanly'. That's not to say that the worker has a chance to complete its work or shut itself down. Multiprocessing will absorb the exceptions and not pass them along. You may terminate a child thread using join(). That is probably the safest way to do it. terminate() will just kill the worker thread immediately without any regard to whether or not it has completed its tasks. I believe multiprocessing uses terminate() as well to kill a daemon thread if the parent thread disappears. join() will, however, block until the task has competed and returned. If you want to continue doing work in the parent thread while the child thread is off doing its thing, then another means of syncing up the parent and children threads is needed. As for allowing children threads to spawn off children of its own using subprocess runs the risk of creating a little army of zombie 'grandchildren' if either the parent or child threads terminate before the subprocess completes and returns. Hope that helps Thanks. That helps tremendously! Jason -- http://mail.python.org/mailman/listinfo/python-list
