Charles-François Natali <neolo...@free.fr> added the comment:
> "Avoid sending very large amounts of data via queues, as you could come up
> against system-dependent limits according to the operating system and whether
> pipes or sockets are used. You could consider an alternative strategy, such
> as writing large data blocks to temporary files and sending just the
> temporary file names via queues, relying on the consumer to delete the
> temporary files after processing."
There's a misunderstanding here: there is absolutely no limit on the
size of objects that can be put through a queue (apart from the host's
memory and the 32-bit limit): the problem is really that you can't
just put an arbitrary buch of data to a queue, and then join it before
making sure other processes will *eventually* pop all the data from
the queue.
I.e., you can't do:
q = Queue()
for i in range(1000000):
q.put(<big obj>)
q.join()
for i in range(10000000):
q.get()
That's because join() will wait until the feeder thread has managed to
write all the items to the underlying pipe/Unix socket, and this might
hang if the underlying pipe/socket is full (which will happen after
one has put around 128K without having popped any item).
That's what's explained here:
It's documented in
http://docs.python.org/library/multiprocessing.html#multiprocessing-programming
:
"""
Joining processes that use queues
Bear in mind that a process that has put items in a queue will wait
before terminating until all the buffered items are fed by the
“feeder” thread to the underlying pipe. (The child process can call
the Queue.cancel_join_thread() method of the queue to avoid this
behaviour.)
This means that whenever you use a queue you need to make sure that
all items which have been put on the queue will eventually be removed
before the process is joined. Otherwise you cannot be sure that
processes which have put items on the queue will terminate. Remember
also that non-daemonic processes will be automatically be joined.
"""
If find this wording really clear, but if someone comes up with a
better - i.e. less technical - wording, go ahead.
----------
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue8426>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
