[issue8037] multiprocessing.Queue's put() not atomic thread wise

Charles-François Natali added the comment: > Modifying an object which is already on a traditional queue can also > change what is received by the other thread (depending on timing). > So Queue.Queue's put() is not "atomic" either. Therefore I do not > believe this behaviour is a bug. Agreed.

[issue8037] multiprocessing.Queue's put() not atomic thread wise

sbt added the comment: I meant Issue 6721 (Locks in python standard library should be sanitized on fork) not 6271. -- ___ Python tracker ___

[issue8037] multiprocessing.Queue's put() not atomic thread wise

sbt added the comment: Modifying an object which is already on a traditional queue can also change what is received by the other thread (depending on timing). So Queue.Queue's put() is not "atomic" either. Therefore I do not believe this behaviour is a bug. However the solution proposed i

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: Please find attached an updated patch (including unit test) for the release27-maint branch. I've run the test_multiprocessing suite against svn revision 86129 of said branch. -- Added file: http://bugs.python.org/file19472/patch_27maint.diff ___

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Ask Solem added the comment: Pickling on put makes sense to me. I can't think of cases where this could break existing code either. I think this may also resolve issue 8323 -- stage: -> unit test needed ___ Python tracker

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: The idea is to pickle the object immediately (i.e. when added) instead of when requested (dequeued). This means that any operations (even deletion) performed on the original object do not make changes to the item in the queue, before it has been dequeued. Wh

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Ask Solem added the comment: aha, no. I see now you use connection.send_bytes instead. Then I can't think of any issues with this patch, but I don't know why it was done this way in the first place. -- ___ Python tracker

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Ask Solem added the comment: AFAICS the object will be pickled twice with this patch. See Modules/_multiprocessing/connection.h: connection_send_obj. -- nosy: +asksol ___ Python tracker

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : -- versions: +Python 3.1 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : Added file: http://bugs.python.org/file16475/queues.diff ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16472/queues.diff ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: Updated patch to include new test (now against release26-maint branch). Verified test_multiprocessing suite passes before (apart from new test) & after change. (Tested under Ubuntu 9.10 64-bit) -- Added file: http://bugs.python.org/file16472/queues.d

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16415/pickle_suggestion_v2.patch ___ Python tracker ___ ___ Python-bugs-l

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: Updated patch to include new test (now against release26-maint branch). Verified test_multiprocessing suite passes before (apart from new test) & after change. (Tested under Ubuntu 9.10 64-bit) -- Added file: http://bugs.python.org/file16473/queues.d

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16473/queues.diff ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : Added file: http://bugs.python.org/file16416/queue_perf.py ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16414/pickle_suggestion.patch ___ Python tracker ___ ___ Python-bugs-list

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: Performance comparison before/after suggested change (adding/removing 1000-item list 1 times). Inline 1.926 1.919 1.907 (bufferPickled = True) 2.138 2.379 2.304 (bufferPickled = True) 1.158 1.151 1.141 (bufferPickled = True) -

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: Alternative suggestion (since SimpleQueue doesn't provide buffering): Allow option to force immediate pickling of the object to be en-queued, i.e. pickling when adding to internal buffer instead of as part of Connection.send() in _feed thread. Does this soun

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : Added file: http://bugs.python.org/file16413/queue_example.py ___ Python tracker ___ ___ Python-bugs-list mailing

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16409/queue_example.py ___ Python tracker ___ ___ Python-bugs-list mailin

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16412/lock_suggestion.patch ___ Python tracker ___ ___ Python-bugs-list m

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: I couldn't see the wood for the trees: If put() is thread-blocking, multiprocessing.Queue is reduced to the same functionality as multiprocessing.queues.SimpleQueue, if I'm not mistaken. So maybe there should be a warning in the documentation that, for multi

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: Hi Meador, I apologise: I really shouldn't have called it a patch. It's just to show one way that I've found fixes the problem (but presumably reduces performance in other cases, hence the suggestion & wait for feedback on the issue). I'm not really experie

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: Quick though (last one I promise): Maybe put() could have an argument to indicate whether to be thread-blocking (i.e. guaranteeing the object to be sent in its current state) or not. The (current) non-thread-blocking mode will have better performance for the

[issue8037] multiprocessing.Queue's put() not atomic thread wise

Vilnis Termanis added the comment: More notes: - Reason for non-atmoic behaviour is due to queue _feeder thread sending the object after the put() method has already exited. Hence changing the object straight after put() can result in the modified object being sent instead. - Reproduced under

[issue8037] multiprocessing.Queue's put() not atomic thread wise

New submission from Vilnis Termanis : If an object, which as been put() in the multiprocessing.Queue is changed immediately after the put() call then changed version may be added to the queue which I assume is not the expected behaviour: >>> from multiprocessing import Queue >>> q = Queue() >>