[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Ronald Oussoren added the comment: Same here, even if I learned more about the implementation of posix_spawn than I should have to know ;-) -- ___ Python tracker ___

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

STINNER Victor added the comment: > Thanks for chiming in Florian, and thanks Pablo for your detailed > investigation. :) I concur, very interesting talk ;-) -- ___ Python tracker _

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Change by Pablo Galindo Salgado : -- resolution: -> rejected stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Gregory P. Smith added the comment: Agreed on not exposing it. It seems obsolete in recent glibc and the older glibc implementations that had it may have made questionable decisions. :) Thanks for chiming in Florian, and thanks Pablo for your detailed investigation. :) --

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

STINNER Victor added the comment: I rely on Florian Weimer who maintains the glibc for Red Hat, and so I agree to close the issue. It seems like Pablo wants to keep the issue open 24h, so I let him close it later ;-) The good news is that calling posix_spawn() with no flag is safe in all gli

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Pablo Galindo Salgado added the comment: Ok, I am going to close the issue and the PR unless someone comments on this issue in 24h saying that we still need to expose the flag and providing an explanation. Thank you everyone for this interesting discussion :) -- ___

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Florian Weimer added the comment: I wouldn't bother with POSIX_SPAWN_USEVFORK on GNU/Linux. Current versions of glibc always use a vfork-style clone call, so there would be a difference on older versions only. But there, the vfork code has subtle bugs, so using POSIX_SPAWN_USEVFORK there i

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Pablo Galindo Salgado added the comment: The removal of POSIX_SPAWN_USEVFORK in glibc was somewhat recent. There is also multiple implementations of posix_spawn in glibc source tree. vfork was introduced in commit 9ad684229e7cf2f0b3c6068b2122701d167a5794 Author: Ulrich Drepper Date: Sun Se

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

STINNER Victor added the comment: I'm discussing with Pablo to see how to use posix_spawn() in the Python subprocess module. IMHO we should consider the specific case of subprocess. What is the expected API? kw = {} if hasattr(os, 'POSIX_SPAWN_USEVFORK'): kw['flags'] = os.POSIX_SPAWN_USEV

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Ronald Oussoren added the comment: Note that the POSIX_SPAWN_USEVFORK may not do anything at this point (see the link in my previous message, if I read the code correctly POSIX_SPAWN_USEVFORK is no longer used in the implementation of posix_spawn(3)). Even if it did do something the user tha

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Gregory P. Smith added the comment: Give all of this, the lesson I'd take away is perhaps that we should just provide the constant in the os module when available at build time (a configure check) and let people who find a need to use it on their system check for its presence and use it in t

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

STINNER Victor added the comment: Oh wow, I didn't expect that exposing a constant would be a source of such deep debate! (I'm not saying that the debate is useless or negative, it's useful and constructive, just I'm surprised how system programming can be hard sometimes!) -- _

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Ronald Oussoren added the comment: I did some more research: * seems to indicate that glibc switched to a different implementation of posix_spawn that uses clone(2) instead of fork()/vfork() to avoid some problems with vfork(). The st

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Ronald Oussoren added the comment: I get the impression that go uses vfork on Linux, not posix_spawn, because go doesn't use libc (based on reading the referenced issue, not on deep knowledge of go and its implementation). I do wonder why glibc's implementation of posix_spawn doesn't use vfo

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Pablo Galindo Salgado added the comment: Some interesting read: Go is using vfork/posix_spawn when possible: https://github.com/golang/go/issues/5838 And it seems that have interesting results: https://about.gitlab.com/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/ --

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Pablo Galindo Salgado added the comment: You still need to allow the flag as being safe or unsafe depends on the user code as glibc cannot know about the details of what is going to be executed. That is the reason they have the flag, so the user can disambiguate if is safe or not. If we don'

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Serhiy Storchaka added the comment: If glibc already uses vfork() if it is safe, I suppose that using vfork() in other cases is unsafe, and we shouldn't support this nonstandard option. -- ___ Python tracker __

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Ronald Oussoren added the comment: vfork() is more dangerous than fork() because the parent and child processes share memory (not copy-on-write, but really the same memory). Whether or not this affects posix_spawn depends on its implementation (to give a very vague statement). Glibc already

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

STINNER Victor added the comment: Serhiy Storchaka: "If this is an optimization, what is the downside of always using vfork()?" I don't know the vfork() function, but you can find articles like: "vfork considered dangerous" (old article of 2012) https://ewontfix.com/7/ But it's unclear to me

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Pablo Galindo Salgado added the comment: The documentation says: The child process is created using vfork(2) instead of fork(2) when either of the following is true: * the spawn-flags element of the attributes object pointed to by attrp contains the GNU-specific flag

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Ronald Oussoren added the comment: Good question. A comment on says that glibc already uses vfork where appropriate, explicitly using the flag may not be necessary. Note that I haven't verified if the

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Serhiy Storchaka added the comment: If this is an optimization, what is the downside of always using vfork()? -- ___ Python tracker ___ ___

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Ronald Oussoren added the comment: @vstinner: another option is to ignore "use_vfork" on platforms that don't have POSIX_SPAWN_USEVFORK. Using vfork or not is primarily a optimisation, ignoring the flag should not result in different behaviour (other than speed). -- nosy: +ronaldouss

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

STINNER Victor added the comment: Gregory: on Linux, it does change the behavior. The parent blocks until the child is spawned. Python should let the developer decide to opt-in. Serhiy: IMHO it's better to make posix_spawn() as dumb as possible, a simple wrapper to the C call. use_vfork=True

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Serhiy Storchaka added the comment: If vfork() is used ob FreeBSD, wouldn't be better to make use_vfork=True the default on FreeBSD and raise an error on use_vfork=False? -- nosy: +serhiy.storchaka ___ Python tracker

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Gregory P. Smith added the comment: Given the FreeBSD default and seeming desirability of vfork for this use case, is there a good reason using vfork could not be the default behavior on any OS that supports it? -- ___ Python tracker

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

STINNER Victor added the comment: I discussed with Pablo (at the CPython sprint) about the case of FreeBSD. On FreeBSD, posix_spawn() always uses vfork, so it can be surprising to get an error when using use_vfork=True. But, the error message is now very explicit: it mentions that a specific

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

STINNER Victor added the comment: We discussed with Gregory P. Smith, and we agreed on the "use_vfork=True" name. He is a native english speaker, so I rely on him :-) Moreover, "use_vfork" is closer to POSIX_SPAWN_USEVFORK constant than "vfork". -- __

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

STINNER Victor added the comment: I suggest to name the parameter "use_vfork", or maybe even "vfork". -- nosy: +vstinner ___ Python tracker ___ ___

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

Change by Pablo Galindo Salgado : -- keywords: +patch pull_requests: +8703 stage: -> patch review ___ Python tracker ___ ___ Python

[issue34663] Support POSIX_SPAWN_USEVFORK flag in posix_spawn

New submission from Pablo Galindo Salgado : In some systems, posix_spawn has an optional flag (POSIX_SPAWN_USEVFORK) that is GNU specific and allows the user to force posix_spawn to spawn the child using vfork instead of fork. This is very beneficial as it gives great speedups compare with no