On 24 Oct 2009, at 21:37, larudwer wrote:
"Brian Quinlan" <br...@sweetapp.com> schrieb im Newsbeitrag
news:mailman.1895.1256264717.2807.python-l...@python.org...
Any ideas why this is happening?
Cheers,
Brian
IMHO your code is buggy. You run in an typical race condition.
consider following part in your code:
def _make_some_processes(q):
processes = []
for _ in range(10):
p = multiprocessing.Process(target=_process_worker, args=(q,))
p.start()
processes.append(p)
return processes
p.start() may start an process right now, in 5 seconds or an week
later,
depending on how the scheduler of your OS works.
Agreed.
Since all your processes are working on the same queue it is -- very
--
likely that the first process got started, processed all the input and
finished, while all the others haven't even got started.
Agreed.
Though your first
process exits, and your main process also exits, because the queue
is empty
now ;).
The main process shouldn't (and doesn't exit) - the _do function exits
(with some processes possibly still running) and the next iteration in
for i in range(100):
_do(i)
is evaluated.
while not q.empty():
pass
If you where using p.join() your main process wourd terminate when
the last
process terminates !
That's an different exit condition!
When you say "your main process would terminate", you mean that the
_do function would exit, right? Because process.join() has nothing to
do with terminating the calling process - it just blocks until process
terminates.
When the main process terminates all the garbage collection fun
happens. I
hope you don't wonder that your Queue and the underlaying pipe got
closed
and collected!
I expected the queue and underlying queue and pipe to get collected.
Well now that all the work has been done, your OS may remember that
someone
sometimes in the past told him to start an process.
Sure, that could happen at this stage. Are you saying that it is the
user of the multiprocessing module's responsibility to ensure that the
queue is not collected in the parent process until all the child
processes using it have exited? Actually, causing the queues to never
be collected fixes the deadlock:
+ p = []
def _do(i):
print('Run:', i)
q = multiprocessing.Queue()
+ p.append(q)
print('Created queue')
for j in range(30):
q.put(i*30+j)
processes = _make_some_processes(q)
print('Created processes')
while not q.empty():
pass
print('Q is empty')
This behavior is counter-intuitive and, as far as I can tell, not
documented anywhere. So it feels like a bug.
Cheers,
Brian
def _process_worker(q):
while True:
try:
something = q.get(block=True, timeout=0.1)
except queue.Empty:
return
else:
print('Grabbed item from queue:', something)
The line
something = q.get(block=True, timeout=0.1)
should cause some kind of runtime error because q is already
collected at
that time.
Depending on your luck and the OS this bug may be handled or not.
Obviously
you are not lucky on OSX ;)
That's what i think happens.
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
