New submission from Brian <br...@merrells.org>:
Despite carefully matching my get() and task_done() statements I would
often trigger "raise ValueError('task_done() called too many times')" in
my multiprocessing.JoinableQueue (multiprocessing/queues.py)
Looking over the code (and a lot of debug logging), it appears that the
issue arises from JoinableQueue.put() not being protected with a locking
mechanism. A preemption after the first line allows other processes to
resume without releasing the _unfinished_tasks semaphore.
The simplest solution seems to be allowing task_done() to block while
waiting to acquire the _unfinished_tasks semaphore.
Replacing:
if not self._unfinished_tasks.acquire(False):
raise ValueError('task_done() called too many times')
With simply:
self._unfinished_tasks.acquire()
This would however remove the error checking provided (given the many
far more subtler error that can be made, I might argue it is of limited
value). Alternately the JoinableQueue.put() method could be better
protected.
----------
components: Library (Lib)
messages: 77806
nosy: merrellb
severity: normal
status: open
title: multiprocessing.JoinableQueue task_done() issue
versions: Python 2.6, Python 2.7, Python 3.0, Python 3.1
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue4660>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
