New submission from Spencer Nelson <swnel...@uw.edu>:
I am writing some software that reads records from a very large file (~hundreds of GB), putting them in an `asyncio.Queue` as it goes, and a chain of consumers handle each record and do stuff over the network. To my surprise, my program runs out of memory because the Queue producer coroutine never yields control. I think (but am not sure) that the asyncio.Queue.put method has no preemption point if the queue is not full; I was using an unbounded Queue, so it was _never_ full, so my coroutine was never unscheduled. I have attached a file with a minimal reproducer. It creates an unbounded queue. A 'publish' task calls `queue.put` from an infinite sequence. A 'subscribe' task calls `queue.get`, and prints each item. Nothing gets printed if I run this, because the `queue.put` never blocks. I would expect that `await queue.put` would occasionally cede execution to any other runnable coroutines, even if the queue is unbounded. ---------- components: asyncio files: never_yields.py messages: 386454 nosy: asvetlov, spenczar, yselivanov priority: normal severity: normal status: open title: asyncio.Queue.put never yields if the queue is unbounded type: behavior versions: Python 3.8 Added file: https://bugs.python.org/file49789/never_yields.py _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue43119> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
