[issue33603] Subprocess Thread handles grow with each call and aren't released until script ends
New submission from GranPrego : On windows 7 / 10 I'm using subprocess to launch a dos cmdline executable and returning the results, which is all working fine. However, each time I make a call, the Python handle count is gradually increasing, jumping up , back a few, then jumping up and so on. All the handles are released when the script exits, but quite often python just hangs after a few hours. If I use process explorer to investigate I can see that python has an increasing number of Thread handles, even though I can see the process being called and cleanly exiting. Unfortunately I'm stuck with the dos executable and it's always a one shot of sending it a single command each time and the script calls it a lot. The executable is just taking a string cmdline and returning a couple of lines of text and then exiting. It only runs for a couple of seconds at most. I've tried two variants of calling the process, I was hoping that the with variant would clean up, but there is no difference. Each handle object that gets left behind has a single reference and a non paged quota of 1192, 0 paged. The script is long running and I've seen the handle count reach 46K. result = "" with Popen ([fcptool, parameters], stdout=PIPE, universal_newlines=True, bufsize=1) as process: for line in process.stdout: result = result + line return result or p = subprocess.run([fcptool, parameters], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True, shell=True).stdout return p I can reproduce this on 3 different machines, 2 windows 7 and one windows 10, all Python 3.6. I can't see a way around this at the moment and as far as I can tell, I'm using the call to subprocess correctly. -- components: Interpreter Core, Windows messages: 317296 nosy: GranPrego, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Subprocess Thread handles grow with each call and aren't released until script ends type: resource usage versions: Python 3.6 ___ Python tracker <https://bugs.python.org/issue33603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33603] Subprocess Thread handles grow with each call and aren't released until script ends
GranPrego added the comment: Process explorer is showing the handles as belonging to the python executable. I can see the cmd process start,then the executable which terminates cleanly. I can see thread handles appearing under the python process, with some terminating, but more green than red, hence the increase. I can post a screenshot tomorrow. Thanks -- ___ Python tracker <https://bugs.python.org/issue33603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33603] Subprocess Thread handles grow with each call and aren't released [Windows]
GranPrego added the comment: I'm now pretty convinced that sounddevice 0.3.11 library is the culprit, which may in turn point to the portaudio library, or CFFI. I make a call to sd.play() just before calling subprocess to run the dos cmd, the timing was such that process explorer would make it look like the leak was occurring during the subprocess call, but isolating the sd.play shows that it causing the two additional Thread handles to be created and never released until the script ends (which could be 1-48 hours or more) Another section of the code was using sd._terminate() and sd._initialize() to work around a buffersize problem with sounddevice and these calls also leak thread handles. I've cut the program down as much as possible and the following now shows the problem without the call to subprocess. You'll need to change the sd.default.device to an appropriate sound card. Thanks for the quick responses. If you're happy that this is the correct analysis of issue then perhaps it could be reclassified as a different component or I can get in touch with the sounddevice author. Regards, Joe -- Added file: https://bugs.python.org/file47613/soundeviceLeaker.py ___ Python tracker <https://bugs.python.org/issue33603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
