[issue38591] Deprecate Process Child Watchers

Kyle Stanley added the comment: > I think so. It will take a long before we remove it though. In that case, it could be a long term deprecation notice, where we start the deprecation process without having a definitive removal version. This will at least encourage users to look towards using

[issue38591] Deprecate Process Child Watchers

Andrew Svetlov added the comment: > So are we at least in agreement for starting with deprecating > FastChildWatcher? I think so. It will take a long before we remove it though. -- ___ Python tracker _

[issue38591] Deprecate Process Child Watchers

Kyle Stanley added the comment: > You have to account also for the thread stack size. I suggest to look at RSS > memory instead. Ah, good point. I believe get_size() only accounts for the memory usage of the thread object, not the amount allocated in physical memory from the thread stack. T

[issue38591] Deprecate Process Child Watchers

Andrew Svetlov added the comment: To be clear: I mean that FastChildWatcher is safe only if all process's code spaws subprocesses by FastChildWatcher. If ProcessPoolExecutor or direct subprocess calls are used the watcher is unsafe. If some C extension spawns new processes on its own (e.g. in

[issue38591] Deprecate Process Child Watchers

STINNER Victor added the comment: I measured the RSS memory per thread: it's around 13.2 kB/thread. Oh, that's way lower than what I expected. Script: # 10: 9636 kB => 9756 kB: +12 kB/thread # 100: 9476 kB = 10796 kB: +13.2 kB/thread # 1000: 9552 kB = 22776 kB: +13.2 kB/thread import os imp

[issue38591] Deprecate Process Child Watchers

STINNER Victor added the comment: > The 64 bytes was measured by `sys.getsizeof(threading.Thread())`, which only > provides a surface level assessment. I believe this only includes the size of > the reference to the thread object. You have to account also for the thread stack size. I suggest

[issue38591] Deprecate Process Child Watchers

Kyle Stanley added the comment: > I understand that there's *some* overhead associated with spawning a new > thread, but from my impression it's not substantial enough to make a > significant impact in most cases. Although I think this still stands to some degree, I will have to rescind the

[issue38591] Deprecate Process Child Watchers

Benjamin Peterson added the comment: FWIW, I started implementing a pidfd-based child process watcher over on #38692. -- nosy: +benjamin.peterson ___ Python tracker ___ __

[issue38591] Deprecate Process Child Watchers

Kyle Stanley added the comment: > But it spawns a new Python thread per process which can be a blocker issue if > a server memory is limited. I understand that there's *some* overhead associated with spawning a new thread, but from my impression it's not substantial enough to make a signifi

[issue38591] Deprecate Process Child Watchers

Kyle Stanley added the comment: > > If asyncio is only run from the main thread, FastChildWatcher is safe, fast > > and has low memory footprint, no? > Unfortunately, no. FastChildWatcher is safe if you can guarantee that no code > executed in asyncio main thread AND thread pools spawn subpr

[issue38591] Deprecate Process Child Watchers

Andrew Svetlov added the comment: My non-LTS Ubuntu also has 5.3 kernel but I'm talking about the oldest supported RHEL/CentOS. That's why pidfd_open() cannot be a single implementation. It's so new; my local man-pages system has not a record about the API yet (but the web has: http://man7.

[issue38591] Deprecate Process Child Watchers

STINNER Victor added the comment: > Regarding pidfd and kqueue -- I love to see pull requests with proposals. Now > nothing exists. The pidfd is available starting from the latest released > Linux 5.3; we need to wait for a decade before all Linux distros adopt it > and we can drop all othe

[issue38591] Deprecate Process Child Watchers

STINNER Victor added the comment: > ThreadedChildWatcher starts a thread per process but has O(1) complexity. But it spawns a new Python thread per process which can be a blocker issue if a server memory is limited. What if you want to spawn 100 processes? Or 1000 processes? What is the memo

[issue38591] Deprecate Process Child Watchers

Kyle Stanley added the comment: > I think FastChildWatcher and SafeChildWatcher should go, ThreadedChildWatcher > should be kept default and MultiLoopChildWatcher is an option where > ThreadedChildWatcher is not satisfactory. Okay, I think I can understand the reasoning here. Do you think th

[issue38591] Deprecate Process Child Watchers

Andrew Svetlov added the comment: ThreadedChildWatcher starts a thread per process but has O(1) complexity. MultiLoopChildWatcher doesn't spawn threads, it can be used safely with asyncio loops spawn in multiple threads. The complexity is O(N) plus no other code should contest for SIG_CHLD s

[issue38591] Deprecate Process Child Watchers

Change by Kyle Stanley : -- title: Deprecating MultiLoopChildWatcher -> Deprecate Process Child Watchers ___ Python tracker ___ ___