On Fri, May 22, 2020, at 10:54 PM, Glyph wrote: > So, pidfd's cool, we should totally use it. Also we should use posix_spawn > and maybe some other stuff too. But I wonder if there's any heuristic we > could use to speed up our current strategy, like ordering the to-reap list by > putting things with no open FDs at the front of it? And optimistically > assuming that once we've found something to reap, maybe we can stop? And > maybe it should run in a cooperator, rather than just blocking the reactor > indefinitely?
Those all sound like reasonable optimizations with little downside to me. > Honestly it had not occurred to me that people were managing 20k+ python > interpreters at a time with spawnProcess (although, yikes, amazing, you've > gotta talk more about this application, and what kind of hardware you're on!) Nonono, that was a synthetic stress test. It did spawn Python processes, but they immediately called `exec sleep` to conserve memory. The result of the test was that the service would run out of memory _well_ before inefficiencies in Twisted's reaping scheme became a problem. ---Tom _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
