On 2018-10-23 06:13, Nathaniel Smith wrote:
On Sun, Oct 21, 2018 at 8:31 PM, Guido van Rossum <[email protected]> wrote:
On Sun, Oct 21, 2018 at 6:08 PM Nathaniel Smith <[email protected]> wrote:
I'm not sure if this is an issue the way Queue is used in practice, but in
general you have to be careful with this kind of circular flow because if
your queue communicates backpressure (which it should) then circular flows
can deadlock.
Nathaniel, would you be able to elaborate more on the issue of backpressure?
I think a lot of people here are not really familiar with the concepts and
its importance, and it changes how you have to think about queues and the
like.
Sure.
Suppose you have some kind of producer connected to some kind of
consumer. If the producer consistently runs faster than the consumer,
what should happen? By default with queue.Queue, there's no limit on
its internal buffer, so if the producer puts, say, 10 items per
second, and the consumer only gets, say, 1 item per second, then the
internal buffer grows by 9 items per second. Basically you have a
memory leak, which will eventually crash your program. And well before
that, your latency will become terrible. How can we avoid this?
[snip]
The purpose of the sentinel is to tell the consumer(s) that there are no
more items, that the producer has finished producing. The sentinel is
the only item in the queue, there will be no more items after it, and
backpressure is not an issue.
_______________________________________________
Python-ideas mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/
