[issue6721] Locks in python standard library should be sanitized on fork

Changes by Forest Bond : -- nosy: +forest_atq ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue6721] Locks in python standard library should be sanitized on fork

Richard Oudkerk added the comment: Lesha, the problems about "magical" __del__ methods you are worried about actually have nothing to do with threading and locks. Even in a single threaded program using fork, exactly the same issues of potential corruption would be present because the object

[issue6721] Locks in python standard library should be sanitized on fork

Vinay Sajip added the comment: > Use file locks in logging, whenever possible. Logging doesn't just log to files, and moreover, also has locks to serialise access to internal data structures (nothing to do with files). Hence, using file locks in logging is not going to magically solve proble

[issue6721] Locks in python standard library should be sanitized on fork

Changes by Tomaž Šolc : -- nosy: -avian ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: 1) I'm totally in favor of making the standard library safe. For that purpose, I think we should do a combination of: a) Use file locks in logging, whenever possible. b) Introduce LockUnsafelyReinitializedAtFork, using a generation counter, or whatever else, which ca

[issue6721] Locks in python standard library should be sanitized on fork

Gregory P. Smith added the comment: subprocess has nothing to do with this bug. subprocess is safe as of Python 3.2 (and the subprocess32 backport for 2.x). Its preexec_fn argument is already documented as an unsafe legacy. If you want to replace subprocess, go ahead, write something new a

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: Actually, we might be able to automatically spawn a safe fork server _only_ when people start mixing threading and subprocess. I'm not totally sure if this would allow us to salvage multiprocessing as well... The tricky bit is that we'd need to proxy into the fork se

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: A slightly more ambitious solution than crashing / deadlocking always is to have Python automatically spawn a "fork server" whenever you start using threads. Then, you would be able to have "subprocess" work cleanly, and not worry about any of this stuff. I don't kn

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: I feel like I'm failing to get my thesis across. I'll write it out fully: == Thesis start == Basic fact: It is an error to use threading locks in _any_ way after a fork. I think we mostly agree on this. The programs we discussing are **inherently buggy**. We disagree

[issue6721] Locks in python standard library should be sanitized on fork

Vinay Sajip added the comment: >> Re "threading locks cannot be used to protect things outside of a >> single process": > The Python standard library already violates this, in that the > "logging" module uses such a lock to protect the file/socket/whatever, > to which it is writing. logging is

[issue6721] Locks in python standard library should be sanitized on fork

Richard Oudkerk added the comment: > conn = MySQLConn() > start_thread1(conn) > start_thread2(conn): > while True: >if os.fork() == 0: # child > raise Exception('doom') # triggers destructor There is no guarantee here that the lock will be held at the time of the fork. So even

[issue6721] Locks in python standard library should be sanitized on fork

Gregory P. Smith added the comment: We could make any later attempt to acquire or release a lock that was reinitialized while it was held raise an exception. Such exception raising behavior should be conditional at lock construction time; some code (such as logging) never wants to deal with o

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: Re "threading locks cannot be used to protect things outside of a single process": The Python standard library already violates this, in that the "logging" module uses such a lock to protect the file/socket/whatever, to which it is writing. If I had a magic wand that

[issue6721] Locks in python standard library should be sanitized on fork

Gregory P. Smith added the comment: threading locks cannot be used to protect things outside of a single process. Any code using them to do that is broken. In your examples you are suggesting a class that wants to do one or more mysql actions within a destructor and worried that the __del__

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: Deadlocks are dandy, but corruption is cruel. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Uns

[issue6721] Locks in python standard library should be sanitized on fork

Gregory P. Smith added the comment: Anyone using a preexec function in subprocess has already declared that they like deadlocks so that isn't an issue. :) -- ___ Python tracker

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: > I think forkall() on Solaris acts like that, but the normal fork() > function does not. Only the thread which performs fork() will survive > in the child process. Sorry, brain fail. A slightly more contrived failure case is this: subprocess.Popen( ..., preexe

[issue6721] Locks in python standard library should be sanitized on fork

Richard Oudkerk added the comment: > a) fork() is called with the DB lock held by thread1. > b) Some time passes before the child gets to exec(). > c) In that time, the child's thread2 gets to doWork(). > d) Simultaneously, the parent's doWork is still running and holding a lock. > e) Thanks to

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: I am really alarmed by the reinit_locks patches. I scanned the comment history, and looked at the patch. I may have missed something, but it looks to me like the basic behavior is this: "After fork(), all locks are replaced by brand-new lock objects that are NOT lock

[issue6721] Locks in python standard library should be sanitized on fork

Richard Oudkerk added the comment: Attached is an updated version of Charles-François's reinit_locks.diff. Changes: * Handles RLock by assuming that if self->count != 0 when we acquire the lock, then the lock must have been reinitialized by PyThread_ReInitLocks(). * Applies existing fork t

[issue6721] Locks in python standard library should be sanitized on fork

Richard Oudkerk added the comment: > > Is there any particular reason not to merge Charles-François's > > reinit_locks.diff? > > > > Reinitialising all locks to unlocked after a fork seems the only sane > > option. > > I agree with this. > I haven't looked at the patch very closely. I think

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: > So what are you suggesting? That a lock of the default type should > raise an error if you try to acquire it when it has been acquired in a > previous process? I was suggesting a way to make 'logging' fork-safe. No more, no less. Does what my previous comment make

[issue6721] Locks in python standard library should be sanitized on fork

Richard Oudkerk added the comment: > (1) Good catch. I suspect that this could be mitigated even if we cared > about LinuxThreads. I haven't looked, but there's got to be a way to > determine if we are a thread or a fork child. Using a generation count would probably work just as well as the

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: This is a reply to: http://bugs.python.org/issue6721#msg151266 Charles-François raises two problems: 1) LinuxThreads have different PIDs per thread. He then points out that LinuxThreads have long been deprecated. 2) you cannot release locks on fork() because that mig

[issue6721] Locks in python standard library should be sanitized on fork

Gregory P. Smith added the comment: going forward with reinit_locks.diff makes sense. I've added comments to it in the code review link. It is "Patch Set 3" -- ___ Python tracker _

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: Should we go forward on this? -- assignee: gregory.p.smith -> stage: test needed -> patch review type: behavior -> enhancement versions: -Python 2.7, Python 3.2 ___ Python tracker

[issue6721] Locks in python standard library should be sanitized on fork

Changes by Vinay Sajip : -- nosy: +vinay.sajip ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: > Is there any particular reason not to merge Charles-François's > reinit_locks.diff? > > Reinitialising all locks to unlocked after a fork seems the only sane option. I agree with this. I haven't looked at the patch very closely. I think perhaps each lock c

[issue6721] Locks in python standard library should be sanitized on fork

sbt added the comment: Is there any particular reason not to merge Charles-François's reinit_locks.diff? Reinitialising all locks to unlocked after a fork seems the only sane option. -- ___ Python tracker ___

[issue6721] Locks in python standard library should be sanitized on fork

sbt added the comment: Attached is a patch (without documentation) which creates an atfork module for Unix. Apart from the atfork() function modelled on pthread_atfork() there is also a get_fork_lock() function. This returns a recursive lock which is held whenever a child process is created

[issue6721] Locks in python standard library should be sanitized on fork

Gregory P. Smith added the comment: A new lock type will NOT solve this. It is ALWAYS okay to clear all thread/threading module locks after a fork. They are and always have been process-local by definition so they are also by definition 100% invalid to any child process. Anyone who has writ

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: > However, extending RLock to provide ForkClearedRLock (this would be used by > logging, i.e.) is quite straighforward. > > The extended class would simply need to record the process ID, in which the > lock was created, and the process ID, which is tr

[issue6721] Locks in python standard library should be sanitized on fork

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue6721] Locks in python standard library should be sanitized on fork

lesha added the comment: Just wanted to say that I spent something like 8 hours debugging a subprocess + threading + logging deadlock on a real production system. I suspected one of my locks at first, but I couldn't find any. The post-fork code was very simple, and I didn't suspect that logg

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: > Anyway, since my view does not seem to resonate with core developers I I'll > give it a rest for now. Well, the problem is that many views have been expressed in this thread, which doesn't help getting a clear picture of what's needed to make progres

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: For the record, turns out there was a bit of misunderstanding. I used the term deprecate above to mean "warn users (through documentation) that they should not use (a feature)" and not in its Python-dev sense of "remove (a feature) after a period of warning". I

[issue6721] Locks in python standard library should be sanitized on fork

sbt added the comment: multiprocessing.util already has register_after_fork() which it uses for cleaning up certain things when a new process (launched by multiprocessing) is starting. This is very similar to the proposed atfork mechanism. Multiprocessing assumes that it is always safe to de

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: Hi Gregory, > Gregory P. Smith added the comment: > No Python thread is ever fork safe because the Python interpreter itself can > never be made fork safe. > Nor should anyone try to make the interpreter itself safe. It is too complex > and effectively impossibl

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: > (BTW: there are religions without "god", so whom shall e.g. i praise for the > GIL?) Guido? ;) -- ___ Python tracker ___

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: > > then multiprocessing is completely brain-damaged and has been > > implemented by a moron. > > Please do not use this kind of language. > Being disrespectful to other people hurts the discussion. So i apologize once again. 'Still i think this shoul

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: P.S.: I have to apologize, it's Tomaž, not Thomas. (And unless i'm mistaken this is pronounced "TomAsch" rather than the english "Tommes", so i was just plain wrong.) --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: > then multiprocessing is completely brain-damaged and has been > implemented by a moron. Please do not use this kind of language. Being disrespectful to other people hurts the discussion. -- ___ Python tracker

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: Um, and just to add: i'm not watching out for anything, and it won't and it can't be me: ?0%0[steffen@sherwood sys]$ grep -F smp CHANGELOG.svn -B3 | grep -E '^r[[:digit:]]+' | tail -n 1 r162 | steffen | 2006-01-18 18:29:58 +0100 (Wed, 18 Jan 2006) | 35

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: If Nir's analysis is right, and Antoines comment pushes me into this direction, (i personally have not looked at that code), then multiprocessing is completely brain-damaged and has been implemented by a moron. And yes, I know this is a bug tracker, and

[issue6721] Locks in python standard library should be sanitized on fork

Gregory P. Smith added the comment: No Python thread is ever fork safe because the Python interpreter itself can never be made fork safe. Nor should anyone try to make the interpreter itself safe. It is too complex and effectively impossible to guarantee. There is no general solution to this,

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: Here is a morning reasoning exercise - please help find the flaws or refine it: 5) Sanitizing worker threads in the multiprocessing module Sanitizing a worker thread in the context of this problem is to make sure it can not create a state that may deadlock anothe

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: Well, my brain did not deadlock, but after spinning on the problem for a while longer, it now thinks Tomaž Šolc and Steffen are right. We should try to fix the multiprocessing module so it does not deadlock single-thread programs and deprecate fork in multi-thread

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: > Would you like to work on a patch to add an atfork mechanism? I will start with an attempt to generate a summary "report" on this rabbit hole of a problem, the views and suggestions raised by people here and what we may expect from atfork approach, its limitatio

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: >> We can't do that, it would break existing code. > > I would argue that such code is already broken. > - that's not necessarily true, if your code is carefully designed - we can't forbid fork() in a multi-threaded application while it's allowed by PO

[issue6721] Locks in python standard library should be sanitized on fork

Tomaž Šolc added the comment: > We can't do that, it would break existing code. I would argue that such code is already broken. > What do you mean by helper threads? multiprocessing uses threads behind the scenes to handle queue traffic and such for individual forked processes. It's somethin

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: > Except for multiprocessing, does anyone know of any other module in the > standard library that uses fork() and threads at the > same time? After some grepping through the source I couldn't find any other > cases. The same problem arises in case of

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: > Except for multiprocessing, does anyone know of any other module in > the standard library that uses fork() and threads at the same time? > After some grepping through the source I couldn't find any other > cases. It's quite common to launch a subprocess from

[issue6721] Locks in python standard library should be sanitized on fork

Tomaž Šolc added the comment: Except for multiprocessing, does anyone know of any other module in the standard library that uses fork() and threads at the same time? After some grepping through the source I couldn't find any other cases. I'm still in favor of just deprecating using fork() on

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: [...] > A caveat is that since Python is an object oriented language it is more > common than with C that code from a higher level module will be invoked by > code from a lower level module, for example by calling an object method that > was over-rid

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: > Sorry, I fail to see how the "import graph" is related to the correct > lock acquisition order. Some locks are created dynamically, for > example. Import dependency is a reasonable heuristic to look into for inter-module locking order. The rational is explained

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: >> - how do you find the correct order to acquire locks in the parent process? > > One option is to use the import graph to determine call order of atfork > handlers. > If a current std library does not fit into this scheme we can possibly fix it > wh

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: > - what would be the API of this atfork() mechanism (with an example of how it > would be used in the library)? The atfork API is defined in POSIX and Gregory P. Smith proposed a Python one above that we can look into. http://pubs.opengroup.org/onlinepubs/0096953

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: > Well, I ping my view that we should: > Could you please detail the following points: - what would be the API of this atfork() mechanism (with an example of how it would be used in the library)? - how do you find the correct order to acquire locks in

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: > > How do you think multiprocessing.Process launches a new process? > > But it's a single piece of code under control and even > multi-OS/multi-architecture test-coverage, not a general purpose > "Joe, you may just go that way and Python will handle it > corre

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: > How do you think multiprocessing.Process launches a new process? But it's a single piece of code under control and even multi-OS/multi-architecture test-coverage, not a general purpose "Joe, you may just go that way and Python will handle it correctly

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: > I would add a big fat note that multiprocessing.Process should be > used instead today, because how about those of us who are not > sophisticated enough to be appointed to standard committees? How do you think multiprocessing.Process launches a new process?

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: My suggestion to this would be that it should be outdated in the same way that Georg Brandl has suggested for changing the default encoding on python-dev [1], and add clear documentation on that, also in respect to the transition phase .. > The problem

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: > I believe that the comp.programming.threads post from > David Butenhof linked above explains why adding atfork() > handlers isn't going to solve this. In Python atfork() handlers will never run from signal handlers, and if I understood correctly, Charles-Franço

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: > I was thinking about a run time warning that is emitted if you call > os.fork() while multiple threads are active. It is ugly, but at least > it tells you you are doing something that will in most cases not work > correctly. The problem is that multiprocessin

[issue6721] Locks in python standard library should be sanitized on fork

Tomaž Šolc added the comment: > You mean a runtime warning? That would be ugly and clumsy. > A warning is probably a good idea, but it should be added somewhere in os.fork() and threading documentation. I was thinking about a run time warning that is emitted if you call os.fork() while multipl

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: Well, I ping my view that we should: 1) Add general atfork() mechanism. 2) Dive into the std lib and add handlers one by one, that depending on case, either do the lock/init thing or just init the state of the library to some valid state in the child. Once this m

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: > The way I see it is that Charles-François' patch trades a possibility of a > deadlock for a possibility of a child process running with inconsistent > states due to forcibly reinitialized locks. > Yeah, that's why I let this stale: that's really an

[issue6721] Locks in python standard library should be sanitized on fork

Tomaž Šolc added the comment: The way I see it is that Charles-François' patch trades a possibility of a deadlock for a possibility of a child process running with inconsistent states due to forcibly reinitialized locks. Personally, I would rather have an occasional deadlock than an occasiona

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: > If there's agreement that the general problem is unsolvable (so fork and > threads just don't get along with each other), what we could attempt is > trying to limit the side effects in the standard library, so that fewest > users as possible are affected by th

[issue6721] Locks in python standard library should be sanitized on fork

Changes by Terry J. Reedy : -- versions: -Python 3.1 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mai

[issue6721] Locks in python standard library should be sanitized on fork

Changes by Tomaž Šolc : -- nosy: +avian ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/m

[issue6721] Locks in python standard library should be sanitized on fork

Giovanni Bajo added the comment: If there's agreement that the general problem is unsolvable (so fork and threads just don't get along with each other), what we could attempt is trying to limit the side effects in the standard library, so that fewest users as possible are affected by this pro

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: @ Nir Aides wrote (2011-05-16 20:57+0200): > Steffen, can you explain in layman's terms? I am the layman here. Charles-François has written a patch for Python which contradicted his own proposal from msg135079, but he seems to have tested a lot so that

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: Steffen, can you explain in layman's terms? On Sun, May 15, 2011 at 8:03 PM, Steffen Daode Nurpmeso wrote: > > @ Charles-François Natali wrote (2011-05-15 01:14+0200): >> So if we really wanted to be safe, the only solution would be to >> forbid fork() in a multi-

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: > Is it possible the following issue is related to this one? It's hard to tell, the original report is rather vague. But the comment about the usage of the maxtasksperchild argument reminds me of issue #10332 "Multiprocessing maxtasksperchild results

[issue6721] Locks in python standard library should be sanitized on fork

Changes by Charles-François Natali : Removed file: http://bugs.python.org/file21991/reinit_locks.diff ___ Python tracker ___ ___ Python-bugs-li

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: Is it possible the following issue is related to this one? http://bugs.python.org/issue10037 - "multiprocessing.pool processes started by worker handler stops working" -- ___ Python tracker

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: @ Charles-François Natali wrote (2011-05-15 01:14+0200): > So if we really wanted to be safe, the only solution would be to > forbid fork() in a multi-threaded program. > Since it's not really a reasonable option But now - why this? The only really acc

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: > a) We know the correct locking order in Python's std libraries so the problem > there is kind of solved. I think that you're greatly under-estimating the complexity of lock ordering. If we were just implementing a malloc implementation protected wit

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: I think that generally it is better to deadlock than corrupt data. > 2) acquiring locks just before fork is probably one of the best way to > deadlock (acquiring a lock we already hold, or acquiring a lock needed > by another thread before it releases its own lock).

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: Hello Steffen, First, thanks for testing this on OS-X: I only have access to Linux systems (I tested both the semaphore and the emulated semaphore paths). If I understand correctly, the patch works fine with the default build option on OS-X. Then, you

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: @ Charles-François Natali wrote (2011-05-13 13:24+0200): > I happily posted a reinit patch I must say in advance that we have implemented our own thread support 2003-2005 and i'm thus lucky not to need to use anything else ever since. So. And of cou

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: > Hi, > Hello Nir, > Option (2) makes sense but is probably not always applicable. > Option (1) depends on being able to acquire locks in locking order, but how > can we determine correct locking order across libraries? > There are indeed a couple p

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: > ...so how can we establish correct (cross library) locking order > during prepare stage? That sounds like a lost battle, if it requires the libraries' cooperation. I think resetting locks is the best we can do. It might not work ok in all cases, but if it can

[issue6721] Locks in python standard library should be sanitized on fork

Steffen Daode Nurpmeso added the comment: @Nir Aides: *thanks* for this link: http://groups.google.com/group/comp.programming.threads/msg/3a43122820983fde You made my day! -- nosy: +sdaoden ___ Python tracker __

[issue6721] Locks in python standard library should be sanitized on fork

Nir Aides added the comment: Hi, There seem to be two alternatives for atfork handlers: 1) acquire locks during prepare phase and unlock them in parent and child after fork. 2) reset library to some consistent state in child after fork. http://pubs.opengroup.org/onlinepubs/009695399/functions

[issue6721] Locks in python standard library should be sanitized on fork

Changes by Nir Aides : -- nosy: +nirai ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/ma

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: Thanks for the explanations. This sounds like an interesting path. > Each thread implementation (thread_pthread.h, thread_nt.h) > would provide a new PyThread_reinit_lock function that would do the > right thing (pthread_mutex_destroy/init, sem_destroy/init, et

[issue6721] Locks in python standard library should be sanitized on fork

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file21875/unnamed ___ Python tracker ___ ___ Python-bugs-list mailing list Uns

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: Please disregard my comment on PyEval_ReInitThreads and _after_fork: it will of course still be necessary, because it does much more than just reinitializing locks (e.g. stop threads). Also, note that both approaches don't handle synchronization primiti

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: >> - what's current_thread_id ? If it's thread_get_ident (pthread_self), >> since TID is not guaranteed to be inherited across fork, this won't >> work > > Ouch, then the approach I'm proposing is probably doomed. > Well, it works on Linux with NPTL, b

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: > - what's current_thread_id ? If it's thread_get_ident (pthread_self), > since TID is not guaranteed to be inherited across fork, this won't > work Ouch, then the approach I'm proposing is probably doomed. > And it's true with every lock in the library code:

[issue6721] Locks in python standard library should be sanitized on fork

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: Oops, for liunxthreads, you should of course read "different PIDs", not "same PID". -- ___ Python tracker ___

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: > Yes, we would need to keep track of the thread id and process id inside > the lock. We also need a global variable of the main thread id after > fork, and a per-lock "taken" flag. > > Synopsis: > >    def _reinit_if_needed(self): >        # Call this

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: > Also, this would imply keeping track of the thread currently owning > the lock, Yes, we would need to keep track of the thread id and process id inside the lock. We also need a global variable of the main thread id after fork, and a per-lock "taken" flag. Sy

[issue6721] Locks in python standard library should be sanitized on fork

Charles-François Natali added the comment: # A lock taken from the current thread should stay taken in the # child process. Note that I'm not sure of how to implement this. After a fork, even releasing the lock can be unsafe, it must be re-initialized, see following comment in glibc's

[issue6721] Locks in python standard library should be sanitized on fork

Gregory P. Smith added the comment: Those tests make sense to me. -- Added file: http://bugs.python.org/file21875/unnamed ___ Python tracker ___Those tests make sense to me.

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: Here is a patch with tests for the issue (some of which fail of course). Do we agree that these tests are right? -- keywords: +patch Added file: http://bugs.python.org/file21874/forklocktests.patch ___ Python tracker

[issue6721] Locks in python standard library should be sanitized on fork

Antoine Pitrou added the comment: I encountered this issue while debugging some multiprocessing code; fork() would be called from one thread while sys.stdout was in use in another thread (simply because of a couple of debugging statements). As a result the IO lock would be already "taken" in

  1   2   >