[issue29828] Allow registering after-fork initializers in multiprocessing

Antoine Pitrou added the comment: Superseded by issue 16500, which has now been resolved! -- resolution: -> duplicate stage: needs patch -> resolved status: open -> closed superseder: -> Allow registering at-fork handlers ___ Python tracker

[issue29828] Allow registering after-fork initializers in multiprocessing

Antoine Pitrou added the comment: The motivation for that decisions might seem a bit secondary, but I was worried that libraries who want to register a multiprocessing-based at-fork handler would always have to pay the import cost for multiprocessing. With the registration logic inside the po

[issue29828] Allow registering after-fork initializers in multiprocessing

Antoine Pitrou added the comment: For those who don't follow both issues: I finally submitted a PR for http://bugs.python.org/issue16500, aka have at-fork handlers that work with all Python-issued fork() calls (including subprocess). -- ___ Python t

[issue29828] Allow registering after-fork initializers in multiprocessing

Antoine Pitrou added the comment: > BTW, why can't you use `pthread_atfork` in numpy? I suppose Numpy could use that, but it's much more involved than registering a Python-level callback. Also it would only work because Numpy actually implements its random generator in C. -- ___

[issue29828] Allow registering after-fork initializers in multiprocessing

Yury Selivanov added the comment: BTW, why can't you use `pthread_atfork` in numpy? -- ___ Python tracker ___ ___ Python-bugs-list mai

[issue29828] Allow registering after-fork initializers in multiprocessing

Antoine Pitrou added the comment: The use case is quite clear here. The specific need to re-seed the Numpy PRNG has already come up in two different projects I work on: Numba and Dask. I wouldn't be surprised if other libraries have similar needs. If you want a reproducible RNG sequence, you

[issue29828] Allow registering after-fork initializers in multiprocessing

Nathaniel Smith added the comment: I think ideally on numpy's end we would reseed iff the RNG was unseeded. Now that I think about it I'm a little surprised that we haven't had more complaints about this, so I guess it's not a super urgent issue, but that would be an improvement over the statu

[issue29828] Allow registering after-fork initializers in multiprocessing

Davin Potts added the comment: Having a read through issue16500 and issue6721, I worry that this could again become bogged down with similar concerns. With the specific example of NumPy, I am not sure I would want its random number generator to be reseeded with each forked process. There are

[issue29828] Allow registering after-fork initializers in multiprocessing

Antoine Pitrou added the comment: That issue seems to have stalled as it seems to have focussed on low-level APIs, and also because it is proposing a new module with the API question that entails. Another possible stance is that os.fork() should be left as-is, as a low-level primitive, and th

[issue29828] Allow registering after-fork initializers in multiprocessing

Yury Selivanov added the comment: Maybe a better way would be to proceed with http://bugs.python.org/issue16500? -- nosy: +yselivanov ___ Python tracker ___ _

[issue29828] Allow registering after-fork initializers in multiprocessing

Changes by Antoine Pitrou : -- nosy: +njs ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.o

[issue29828] Allow registering after-fork initializers in multiprocessing

New submission from Antoine Pitrou: Currently, multiprocessing has hard-coded logic to re-seed the Python random generator (in the random module) whenever a process is forked. This is present in two places: `Popen._launch` in `popen_fork.py` and `serve_one` in `forkserver.py` (for the "fork"