Joongi Kim <m...@daybreaker.info> added the comment:
This particular experience, https://github.com/lablup/backend.ai-agent/pull/331, has actually motivated me to suggest PersistentTaskGroup. The program subscribes the event stream of Docker daemon using aiohttp as an asyncio task, and this should be kept running throughout the whole application lifetime. I first applied aiotools.TaskGroup to ensure shutdown of spawned event handler tasks, but I missed that it cancels all sibling tasks if one of the spawned tasks bubbles up an unhandled exception. This has caused silent termination of the subscriber task and led to a bug. We could debug this issue by inspecting aiomonitor and checking the existence of this task. After this issue, I began to think we need a proper abstraction of a long-running task group (NOTE: the task group is long-running. The lifetime of internal tasks does not matter). Another case is that https://github.com/lablup/backend.ai/issues/330. One of our customer site has suffered from excessive CPU usage by our program. We could identify the issue by aiomonitor, and the root cause was the indefinite accumulation of peridoically created asyncio tasks to measure the disk usage of user directories, when there are too many files in them. Since the number of tasks have exceeded 10K, it was very difficult to group and distinguish individual asyncio tasks in aiomonitor. I thought that it would be nice if we could group such tasks into long-running groups and view task statistics separately. ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue46843> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
