[issue40860] Exception in multiprocessing/context.py under load

Change by Irit Katriel : -- resolution: -> duplicate stage: -> resolved status: open -> closed superseder: -> Multiprocessing docs don't describe thread-safety ___ Python tracker __

[issue40860] Exception in multiprocessing/context.py under load

Irit Katriel added the comment: I agree that it would be better not to lock join, and instead manage each process from one thread. I think this ticket can be closed as a duplicate of issue40815. -- ___ Python tracker

[issue40860] Exception in multiprocessing/context.py under load

Arkady added the comment: "documentation of thread safety" I find it surprising that a module called "multiprocessing" has not a thread safe API. If this is inevitable, I guess that's the life. I expect nothing less that a bold bright red font at the top of the document page. Protecting

[issue40860] Exception in multiprocessing/context.py under load

Irit Katriel added the comment: There is an open ticket to improve documentation of thread safety of the multiprocessing module module: https://bugs.python.org/issue40815 Is there anything remaining to do on this ticket? -- ___ Python tracker

[issue40860] Exception in multiprocessing/context.py under load

Arkady added the comment: I have switched to os.fork() I am doing something like this https://gist.github.com/larytet/3ca9f9a32b1dc089a24cb7011455141f -- ___ Python tracker _

[issue40860] Exception in multiprocessing/context.py under load

Irit Katriel added the comment: The source code for the Process class is here: https://github.com/python/cpython/blob/master/Lib/multiprocessing/process.py You can see that join and start both modify the global, non thread safe _children set. I'm guessing this is where you're seeing interfer

[issue40860] Exception in multiprocessing/context.py under load

Arkady added the comment: There is a memory leak every time call to join() fails (is https://bugs.python.org/issue37788 relevant?) -- ___ Python tracker ___ _

[issue40860] Exception in multiprocessing/context.py under load

Arkady added the comment: A workaround is to synchronize the call to Process.start() diff --git a/main.py b/main.py index d09dc53..49d68f0 100644 --- a/main.py +++ b/main.py @@ -26,17 +26,24 @@ def load_cpu(deadline): while time.time() - start < 0.2*deadline: math.pow(random.rand

[issue40860] Exception in multiprocessing/context.py under load

Change by Ned Deily : -- nosy: +davin, pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.py

[issue40860] Exception in multiprocessing/context.py under load

Arkady added the comment: Update. I have reproduced the problem in the code not calling Process.join() at all. It require more time and more load, but eventually Process.start() crashes. Posted a question on https://stackoverflow.com/questions/62276345/call-to-pythons-mutliprocessing-process

[issue40860] Exception in multiprocessing/context.py under load

Arkady added the comment: Is there any news about? This 50 lines sample reproduces the problem https://github.com/larytet-py/multiprocess Please let me know if more information is needed. -- ___ Python tracker

[issue40860] Exception in multiprocessing/context.py under load

Arkady added the comment: The problem is likely in the call to multiprocessing.Process.join() with timeout. If I use timeout=None the code works. -- ___ Python tracker ___ __

[issue40860] Exception in multiprocessing/context.py under load

larytet added the comment: This code reproduces the problem https://github.com/larytet-py/multiprocess I assume that my use of join() is not correct. -- ___ Python tracker ___

[issue40860] Exception in multiprocessing/context.py under load

New submission from Arkady M : I am running an HTTP server (socketserver.ThreadingMixIn, http.server.HTTPServer) in a Docker container (FROM ubuntu:19.10) Occasionally I get an exception: Exception happened during processing of request from ('172.17.0.1', 35756) Traceback (most recent call la