[issue6721] Locks in the standard library should be sanitized on fork

Change by Kyle Evans : -- nosy: +kevans nosy_count: 28.0 -> 29.0 pull_requests: +21627 pull_request: https://github.com/python/cpython/pull/22651 ___ Python tracker ___

[issue6721] Locks in the standard library should be sanitized on fork

Change by Andrés Delfino : -- nosy: -adelfino ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue6721] Locks in the standard library should be sanitized on fork

Change by Stefan Behnel : -- nosy: -scoder ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue6721] Locks in the standard library should be sanitized on fork

Change by Stefan Behnel : -- pull_requests: -21519 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue6721] Locks in the standard library should be sanitized on fork

Change by Stefan Behnel : -- nosy: +scoder nosy_count: 29.0 -> 30.0 pull_requests: +21519 pull_request: https://github.com/python/cpython/pull/22474 ___ Python tracker ___ _

[issue6721] Locks in the standard library should be sanitized on fork

Change by Andrés Delfino : -- nosy: +adelfino nosy_count: 28.0 -> 29.0 pull_requests: +21315 pull_request: https://github.com/python/cpython/pull/22205 ___ Python tracker ___ ___

[issue6721] Locks in the standard library should be sanitized on fork

Change by mohamed koubaa : -- nosy: +koubaa nosy_count: 27.0 -> 28.0 pull_requests: +21218 pull_request: https://github.com/python/cpython/pull/21986 ___ Python tracker ___

[issue6721] Locks in the standard library should be sanitized on fork

Change by Jakub Kulik : -- pull_requests: -21197 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail

[issue6721] Locks in the standard library should be sanitized on fork

Change by Jakub Kulik : -- nosy: -kulikjak ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue6721] Locks in the standard library should be sanitized on fork

Change by Jakub Kulik : -- nosy: +kulikjak nosy_count: 27.0 -> 28.0 pull_requests: +21197 pull_request: https://github.com/python/cpython/pull/22040 ___ Python tracker ___ _

[issue6721] Locks in the standard library should be sanitized on fork

Change by Gregory P. Smith : -- pull_requests: -21110 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:/

[issue6721] Locks in the standard library should be sanitized on fork

Change by Raymond Hettinger : -- nosy: +rhettinger nosy_count: 26.0 -> 27.0 pull_requests: +21110 pull_request: https://github.com/python/cpython/pull/21994 ___ Python tracker _

[issue6721] Locks in the standard library should be sanitized on fork

Change by Jesse Farnham : -- nosy: +jesse.farnham ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail

[issue6721] Locks in the standard library should be sanitized on fork

STINNER Victor added the comment: See also bpo-25920: PyOS_AfterFork should reset socketmodule's lock. -- ___ Python tracker ___ ___

[issue6721] Locks in the standard library should be sanitized on fork

Deomid Ryabkov added the comment: https://bugs.python.org/issue40442 is a fresh instance of this, entirely self-inflicted. -- nosy: +rojer ___ Python tracker ___ _

[issue6721] Locks in the standard library should be sanitized on fork

Antoine Pitrou added the comment: Related issue: https://bugs.python.org/issue40399 """ IO streams locking can be broken after fork() with threads """ -- ___ Python tracker ___

[issue6721] Locks in the standard library should be sanitized on fork

STINNER Victor added the comment: I created bpo-40089: Add _at_fork_reinit() method to locks. -- ___ Python tracker ___ ___ Python-b

[issue6721] Locks in the standard library should be sanitized on fork

Change by Hugh Redelmeier : -- nosy: +hugh ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue6721] Locks in the standard library should be sanitized on fork

Gregory P. Smith added the comment: Thanks for the debugging details! I've filed https://bugs.python.org/issue36533 to specifically track this potential regression in the 3.7 stable branch. lets carry on there where the discussion thread isn't too long for bug tracker sanity. --

[issue6721] Locks in the standard library should be sanitized on fork

Change by A. Jesse Jiryu Davis : -- nosy: -emptysquare ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:

[issue6721] Locks in the standard library should be sanitized on fork

cagney added the comment: Below is a backtrace from the deadlock. It happens because the logging code is trying to acquire two per-logger locks; and in an order different to the random order used by the fork() handler. The code in question has a custom class DebugHandler(logging.Handler). T

[issue6721] Locks in the standard library should be sanitized on fork

cagney added the comment: > acquiring locks before fork in the thread doing the forking and releasing > them afterwards is always the safe thing to do. It's also an easy way to cause a deadlock: - register_at_fork() et.al. will cause per-logger locks to be acquired before the global lock (t

[issue6721] Locks in the standard library should be sanitized on fork

Gregory P. Smith added the comment: I'd start with faulthandler.register with all_threads=True and see if that gives you what you need. https://docs.python.org/3/library/faulthandler.html -- ___ Python tracker

[issue6721] Locks in the standard library should be sanitized on fork

cagney added the comment: > Does your code use any C code that forks on its own without properly calling > the C Python PyOS_BeforeFork(), PyOS_AfterFork_Parent(), and > PyOS_AfterFork_Child() APIs? No. Is there a web page explaining how to pull a python backtrace from all the threads runn

[issue6721] Locks in the standard library should be sanitized on fork

Gregory P. Smith added the comment: We need a small test case that can reproduce your problem. I believe https://github.com/python/cpython/commit/3b699932e5ac3e76031bbb6d700fbea07492641d to be correct. acquiring locks before fork in the thread doing the forking and releasing them afterward

[issue6721] Locks in the standard library should be sanitized on fork

cagney added the comment: I suspect 3b699932e5ac3e7 is causing a hang in libreswan's kvmrunner.py on Fedora. Looking at the Fedora RPMs: python3-3.7.0-9.fc29.x86_64 didn't contain the fix and worked python3-3.7.1-4.fc29.x86_64 reverted the fix (for anaconda) and worked python3-3.7.2-4.fc29.x

[issue6721] Locks in the standard library should be sanitized on fork

STINNER Victor added the comment: > New changeset 3b699932e5ac3e76031bbb6d700fbea07492641d by Gregory P. Smith > (Miss Islington (bot)) in branch '3.7': > bpo-6721: Hold logging locks across fork() (GH-4071) (#9291) It seems like this change caused a regression in the Anaconda installer of F

[issue6721] Locks in the standard library should be sanitized on fork

Gregory P. Smith added the comment: New changeset 3b699932e5ac3e76031bbb6d700fbea07492641d by Gregory P. Smith (Miss Islington (bot)) in branch '3.7': bpo-6721: Hold logging locks across fork() (GH-4071) (#9291) https://github.com/python/cpython/commit/3b699932e5ac3e76031bbb6d700fbea07492641d

[issue6721] Locks in the standard library should be sanitized on fork

Change by miss-islington : -- pull_requests: +8722 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue6721] Locks in the standard library should be sanitized on fork

Gregory P. Smith added the comment: New changeset 19003841e965bbf56fd06824d6093620c1b66f9e by Gregory P. Smith in branch 'master': bpo-6721: Hold logging locks across fork() (GH-4071) https://github.com/python/cpython/commit/19003841e965bbf56fd06824d6093620c1b66f9e -- _

[issue6721] Locks in the standard library should be sanitized on fork

Olivier Chédru added the comment: FWIW, I encountered the same kind of issue when using the mkstemp() function: under the hood, it calls gettempdir() and this one is protected by a lock too. Current thread 0x7ff10231f700 (most recent call first): File "/usr/lib/python3.5/tempfile.py", l

[issue6721] Locks in the standard library should be sanitized on fork

Antoine Pitrou added the comment: Actually, we already have a doubly-linked list of buffered IO objects (for another purpose), so we can reuse that and register a single set of global callbacks. -- ___ Python tracker

[issue6721] Locks in the standard library should be sanitized on fork

Gregory P. Smith added the comment: logging is pretty easy to deal with so I created a PR. bufferedio.c is a little more work as we either need to use the posixmodule.c os.register_at_fork API or expose it as an internal C API to be able to call it to add acquires and releases around the buff

[issue6721] Locks in the standard library should be sanitized on fork

Change by Gregory P. Smith : -- pull_requests: +4042 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue6721] Locks in the standard library should be sanitized on fork

Antoine Pitrou added the comment: Oh, I forgot that IO buffered objects also have a lock. So we would have to special-case those as well, unless we take the generic approach... A problem with the generic approach is that it would leave higher-level synchronization objects such as RLock, Even

[issue6721] Locks in the standard library should be sanitized on fork

Antoine Pitrou added the comment: I think we should somehow move forward on this, at least for logging locks which can be quite an annoyance. There are two possible approaches: - either a generic mechanism as posted by sbt in reinit_locks_2.diff - or a logging-specific fix using os.register_at

[issue6721] Locks in the standard library should be sanitized on fork

Daniel Birnstiel added the comment: While having to deal with this bug for a while I have written a small library using `pthread_atfork`: https://github.com/Birne94/python-atfork-lock-release It allows registering atfork-hooks (similar to the ones available by now) and frees the stdout/stderr

[issue6721] Locks in the standard library should be sanitized on fork

Gregory P. Smith added the comment: http://bugs.python.org/issue16500 added the os.register_at_fork() API which may be usable for this. -- ___ Python tracker ___

[issue6721] Locks in the standard library should be sanitized on fork

Changes by Gregory P. Smith : -- versions: +Python 3.7 -Python 3.5 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe

[issue6721] Locks in the standard library should be sanitized on fork

Daniel Birnstiel added the comment: Currently using Python 3.6.0 (default, Mar 4 2017, 12:32:34) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin > So, somehow the print() statement is blocking, which I have /no/ idea how to > go about debugging. I assume there's a lock /i

[issue6721] Locks in the standard library should be sanitized on fork

Changes by Camilla Montonen : -- nosy: +Winterflower ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue6721] Locks in the standard library should be sanitized on fork

Changes by Davin Potts : -- nosy: +davin ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org

[issue6721] Locks in the standard library should be sanitized on fork

STINNER Victor added the comment: I suggest to close the issue as WONT FIX. Python code base is huge and Python depends on a lot of external code. We cannot warranty anything. It might be possible to track all kinds of locks with an infinite time. But I'm not sure that it's worth it. It is poss

[issue6721] Locks in the standard library should be sanitized on fork

Connor Wolf added the comment: > Python 3.5.1+ (default, Mar 30 2016, 22:46:26) Whatever the stock 3.5 on ubuntu 16.04 x64 is. I've actually been running into a whole horde of really bizarre issues related to what I /think/ is locking in stdout. Basically, I have a context where I have thous

[issue6721] Locks in the standard library should be sanitized on fork

Gregory P. Smith added the comment: My intent is not to block anything. I'm Just explaining why I'm not motivated to spend much time on this issue myself. Others are welcome to. subprocess is not related to this issue, it has been fixed for use with threads (in 3.2 and higher) with an extrem

[issue6721] Locks in the standard library should be sanitized on fork

lesha added the comment: On a moment's reflection, a lot of my prior comment is wrong. Sorry about that. - glog does not, that I know of, sanitize locks on fork. You just shouldn't log after fork but before exec. - Using `pthread_atfork` to clean up the `logging` lock might be enough to make

[issue6721] Locks in the standard library should be sanitized on fork

lesha added the comment: I work on a multi-million-line C++ codebase that uses fork() from multithreaded programs all over the place. We use `glog` ubiquitously. This bug here that spans 6 years and has affected dozens of people (conservatively) simply does not exist for us. That is because

[issue6721] Locks in the standard library should be sanitized on fork

Connor Wolf added the comment: Arrrgh, s/threading/multiprocessing/g in my last message. -- ___ Python tracker ___ ___ Python-bugs-list

[issue6721] Locks in the standard library should be sanitized on fork

Connor Wolf added the comment: > IMNSHO, working on "fixes" for this issue while ignoring the larger > application design flaw elephant in the room doesn't make a lot of sense. I understand the desire for a canonically "correct" fix, but it seems the issue with fixing it "correctly" has lead t

[issue6721] Locks in the standard library should be sanitized on fork

Gregory P. Smith added the comment: For me the momentum on fixing these things has stalled because I no longer work on any code that runs into this. There is a fundamental problem: You cannot safely use threading and os.fork() in the same application per POSIX rules. So even if the standard

[issue6721] Locks in the standard library should be sanitized on fork

Connor Wolf added the comment: Is anything happening with these fixes? This is still an issue (I'm running into it now)? -- nosy: +Connor.Wolf ___ Python tracker ___

[issue6721] Locks in the standard library should be sanitized on fork

Changes by A. Jesse Jiryu Davis : -- nosy: +emptysquare ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue6721] Locks in the standard library should be sanitized on fork

Changes by Nir Soffer : -- nosy: +nirs ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/m

[issue6721] Locks in the standard library should be sanitized on fork

Changes by Maries Ionel Cristian : -- nosy: +ionel.mc ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue6721] Locks in the standard library should be sanitized on fork

Changes by Dan O'Reilly : -- nosy: +dan.oreilly ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue6721] Locks in the standard library should be sanitized on fork

Changes by Tshepang Lekhonkhobe : -- nosy: +tshepang ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue6721] Locks in the standard library should be sanitized on fork

Changes by Tshepang Lekhonkhobe : -- title: Locks in python standard library should be sanitized on fork -> Locks in the standard library should be sanitized on fork versions: +Python 3.5 -Python 3.3 ___ Python tracker