On Sun, May 17, 2020, at 12:22 PM, Glyph wrote:
>> In the one case that may be complicated I think that pinning Twisted will be
>> acceptable, since the service in question is a process supervisor without
>> direct external exposure.
> Well in that case I'll just do my best to add lots of goodies (posix_spawn
> support!) that will make you want to upgrade that to current twisted and
> python 3.8 :).
Oh it's not for lack of wanting. :-)
The service in question is actually spawning Python processes, so posix_spawn
support doesn't seem likely to help much: fork() + exec() is a drop in the
bucket compared to Python imports, particularly in Python 3 (I'm hoping to
explore oxidized-importer [1] to help with this at some point).
If looking improve process management in Twisted, I think that the
lowest-hanging fruit is to watch children with pidfd, rather than reaping all
children on SIGCHLD. This O(n) reaping --- one waitpid() call per running
process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000
processes or so.
---Tom
[1]: https://pypi.org/project/oxidized-importer/
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
