[issue8800] add threading.RWLock

Andrew Svetlov added the comment: Thanks for sharing, your project looks viable! -- ___ Python tracker ___ ___ Python-bugs-list mail

[issue8800] add threading.RWLock

Éric Larivière added the comment: Hi, Sorry to wake up a 10 years old discussion But I think that you might be interested in the following Python package that I created and maintain since few years now: https://pypi.org/project/readerwriterlock/ 1- It implements the three type of read

[issue8800] add threading.RWLock

Change by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.

[issue8800] add threading.RWLock

Nathaniel Smith added the comment: Side note, in case anyone else finds themselves looking for the MSDN magazine article referenced above: it appears to be in the June 2007 issue, titled "CONCURRENCY: Synchronization Primitives New To Windows Vista". For the moment at least, this link seems t

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Seems to have fizzled out due to the intense amount of bikeshedding required. -- ___ Python tracker ___ _

[issue8800] add threading.RWLock

Ofek Lev added the comment: What is the status of the patch? -- nosy: +Ofekmeister ___ Python tracker ___ ___ Python-bugs-list mailing

[issue8800] add threading.RWLock

Changes by Dan O'Reilly : -- nosy: +dan.oreilly ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pyt

[issue8800] add threading.RWLock

Changes by Vladimir Rutsky : -- nosy: +vrutsky ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: Le vendredi 05 octobre 2012 à 09:51 +, Kristján Valur Jónsson a écrit : > Then I started to wonder if it were appropriate to use a Barrier in > the Condition variable tests, particularly given that the former is > implemented by way of the latter :) Indeed,

[issue8800] add threading.RWLock

Changes by Kristján Valur Jónsson : -- Removed message: http://bugs.python.org/msg172065 ___ Python tracker ___ ___ Python-bugs-list ma

[issue8800] add threading.RWLock

Changes by Kristján Valur Jónsson : Removed file: http://bugs.python.org/file27430/socketserver.patch ___ Python tracker ___ ___ Python-bugs-li

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: Kristjan: you seem to have attached socketserver.patch to the wrong issue. -- ___ Python tracker ___ ___

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: > although I'd prefer a BSD errno example, such as ECONNRESET, instead of a > winsock one Are you referring to the comment where I mention ECONNABORTED? That is a regular unix error (windows version is WSAECONNABORTED). This error occurs when the local

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Thanks Richard. I wonder if the issues with the multiprocessing tests can be fixed by making use of Barriers? One of the reasons I introduced Barriers into the lib originally was my alarm at seeing the various springklings of different _wait() calls in

[issue8800] add threading.RWLock

Changes by Richard Oudkerk : Removed file: http://bugs.python.org/file27421/rwlock-sbt.patch ___ Python tracker ___ ___ Python-bugs-list mailin

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: Fixed patch because I didn't test on Unix... -- Added file: http://bugs.python.org/file27422/rwlock-sbt.patch ___ Python tracker ___ ___

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: Attached is a new version of Kristjan's patch with support for managers. (A threading._RWLockCore object is proxied and wrapped in a local instance of a subclass of threading.RWLock.) Also I made multiprocessing.RWLock.__init__() use multiprocessing.util.re

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Ok, here is a new patch. It takes into account various comments and suggestions: 1) The interface is java-like. a single RWLock instance only has attributes reader_lock and writer_lock. 2) Since the "owning" array needs only be process local, the same

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: By the way, the fact that acquire_read() is documented as "Acquire the lock in shared mode" and acquire_write() as "Acquire the lock in exclusive mode" hints that perhaps RWLock is not the right name ;) -- ___ Pytho

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: I don't like the API. Why should I write "rwlock.acquire_read()" instead of "rwlock.reader.acquire()"? There should be one obvious way to do it. The fact that "reader_lock" and "writer_lock" return a new object every time is a bit suboptimal, IMO. Also, RWLoc

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Excellent point, I hadn't thought of that! Yes, it is is sufficient to test if _I_ am in the list. I'll make the necessary changes. That will make the thread/process implementation virtually identical. -- _

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: > Multiprocessing: Because there is no way I know to share a list of > owning thread ids, this version is more limited Why do you need a *shared* list? I think it should be fine to use a per-process list of owning thread ids. So the current thread owns th

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: I admit that I kind of like Java's approach to this. First off, they define an interface, ReadWriteLock: http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/ReadWriteLock.html There, they also discuss the different choices an impleme

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Ah, you are implementing an FIFO lock. That should have been made clear. I see it now that you grant the lock in the order that the acquisition is attempted. Ok, this is fine, but with one important caveat: Explicit handoff such as that can suffer fro

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: This amount of code provides recursion, context managers, condition variable compatibility, timeout functionality, error checking and conformance with the unit tests. The actual locking code is encapsulated in the three functions acquire_read(), acqui

[issue8800] add threading.RWLock

Sebastian Noack added the comment: @Kristján: Uhh, that is a huge amount of code, more than twice as much (don't counting tests) as my implementation, to accomplish the same. And it seems that there is not much code shared between the threading and multiprocessing implementation. And for what?

[issue8800] add threading.RWLock

Sebastian Noack added the comment: Exactly, with my implemantation "the lock acquired first will be granted first". There is no way that either shared nor exclusive locks can starve, and therefore it should satisfy all use cases. Since you can only share simple datastructures like integers acr

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Oh, I forgot to mention: Once one gets into the domain of allowing such niceties as writer priority, surely you can agree that the implementation of both locking modes belongs in the same class instance. That is just plain good coding practice, allow

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Here is a new patch. it is complete with: threading implementation and tests multiprocessing implementation and tests. Let's leave the naming bikeshedding a bit and focus on some practical aspects: 1) The threading version contains a RWLock and a FairR

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: > The unlock operation is the same, so now you have to arbitrarily pick one > of the "lockd" and chose release(). That depends on the implementation. In the three implementations on http://en.wikipedia.org/wiki/Readers-writers_problem the unlock operate

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: > I think you got that argument backwards. The simple greedy policy you > implement works well provided there are not too many readers. Otherwise, > the writers will be starved, since they have to wait for an oppertune > moment when no readers are active to

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: > I have implemented the simplest possible acquisition order. > The lock acquired first will be granted first. Without that (or a more > advanced policy) in applications with concurrent threads/processes > that are heavily using the shared lock, the exclu

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: > We've already departed from that. Our Lock is nothing like a mutex, for > example (it's more of a binary semaphore). This is not by nature of good design, but an accident. C python needed both mutex and signaling ability and decided that a single non

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: > You are not restricted to the context manager model. Just use > selock.shared.acquire() or selock.exclusive.acquire(). The unlock operation is the same, so now you have to arbitrarily pick one of the "lockd" and chose release(). Why take a construc

[issue8800] add threading.RWLock

Charles-François Natali added the comment: > The page also mentions a seqlock which looks interesting to me as > it's fast for few writers with lots of readers. A seqlock is suitable for consistent views of simple data structures (e.g. a counter in the Linux kernel), but it won't fly for a high

[issue8800] add threading.RWLock

Sebastian Noack added the comment: Thanks, but as I already said there are a lot of implementations for shared/exclusive lock that can be acquired from different threads. But we need with threading as well as with multiprocessing. And by the way POSIX is the standard for implementing UNIX-like

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: > A RW lock is part of POSIX threads [1]. It's usually a good idea to > either use POSIX functions or to mimic their behavior. After all POSIX > is an industry standard. We've already departed from that. Our Lock is nothing like a mutex, for example (it's more o

[issue8800] add threading.RWLock

Christian Heimes added the comment: A RW lock is part of POSIX threads [1]. It's usually a good idea to either use POSIX functions or to mimic their behavior. After all POSIX is an industry standard. Boost and Java have several lock and rw lock implementations. Wikipedia [2] is a good startin

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: On the naming front: shorthands like "Shrd" and "Excl" are a bit frown upon. Since "SharedExclusiveLock" is on the long side, I would suggest calling the API "SELock". -- ___ Python tracker

[issue8800] add threading.RWLock

Sebastian Noack added the comment: I would love to see how other people would implement a shared/exclusive lock that can be acquired from different processes. However it really seems that nobody did it before. If you know a reference implementation I would be more than happy. There are plenty

[issue8800] add threading.RWLock

Christian Heimes added the comment: I wonder, why are you creating your own algorithm here? There must be plenty of reference implementations that are already used in production code. Don't be a shamed to copy a Java implementation! :) The entire threading module is a rip-off of the Java threa

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: My previous comment applied to Sebastian's first patch. The second seems to fix the issue. -- ___ Python tracker ___ _

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: I think Sebastian's algorithm does not correctly deal with the non-blocking case. Consider the following situation: * Thread-1 successfully acquires exclusive lock. Now num_got_lock == 1. * Thread-2 blocks waiting for shared lock. Will block until (num_g

[issue8800] add threading.RWLock

Sebastian Noack added the comment: > If you want to argue it this way, I counter that the attributes > "shared" and "exclusive" apply to the type of "access to the > protected object" you are talking about, and yet, the name suggest > that they are attributes of the lock itself. A lock's sole pu

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Perhaps I should have pointed out, for Sebastian's benefit, that my second patch uses "timeout" rather than "blocking" since that is the new black in python 3. Also, I think the "threading" implementation shows clearly the problem of having two indepen

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: > With this, you are stuck with employing a context manager model only. > You loose the flexibility to do explicit acquire_read() or > acquire_write(). You are not restricted to the context manager model. Just use selock.shared.acquire() or selock.exclusiv

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: > @richard: I'm sorry, but both of my patches contain changes to > 'Lib/threading.py' and can be applied on top of Python 3.3.0. So can you > explain what do you mean, by missing the changes to threading.py? I was reading the Rietveld review page http://

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: >> shared/exclusive -> abstract description of what it is If you want to argue it this way, I counter that the attributes "shared" and "exclusive" apply to the type of "access to the protected object" you are talking about, and yet, the name suggest th

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: > Personally, I would prefer to make the shared and exclusive locks > attributes of the same object, so one could do > >with selock.shared: > ... > >with selock.exclusive: > ... > Please note, the "same object" could simply be a name

[issue8800] add threading.RWLock

Sebastian Noack added the comment: @richard: I'm sorry, but both of my patches contain changes to 'Lib/threading.py' and can be applied on top of Python 3.3.0. So can you explain what do you mean, by missing the changes to threading.py? -- ___ Pytho

[issue8800] add threading.RWLock

Sebastian Noack added the comment: Yes, you could also look at the shared/exclusive lock as one lock with different states. But this approach is neither more common, have a look at Java's ReadWriteLock [1] for example, which works just like my patch does, except that a factory is returned inst

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: @Sebastian: Both your patch sets are missing the changes to threading.py. -- ___ Python tracker ___

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: Ok, since we're bikeshedding, I like Richard's proposal best: > Personally, I would prefer to make the shared and exclusive locks > attributes of the same object, so one could do > >with selock.shared: > ... > >with selock.exclusive: > ...

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: >A shared/exclusive lock isn't one lock but two locks, which are >synchronized, >but must be acquired separately. Similar to a pipe, >which isn't one file, but >one file connected to another file that >reads whatever you have written into >the first fi

[issue8800] add threading.RWLock

Sebastian Noack added the comment: I was just waiting for a comment pointing out, that my patch comes without tests. :) Note that we are still discussing the implementation and this patch is just a proof of concept. And since the way it is implemented and the API it provides could still change

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: I can't say that I'm that familiar with multiprocessing to comment on that in particular. But I do find your approach strange, to create two "lock-like" objects, in stead of the more familiar construct of having a "RWLock" (this is known from other lan

[issue8800] add threading.RWLock

Richard Oudkerk added the comment: > I've added a new patch, that implements a shared/exclusive lock as > described in my comments above, for the threading and multiprocessing > module. The patch does not seem to touch the threading mode and does not come with tests. I doubt the multiprocess

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Yes, any "blocking" on the internal lock is not semantic blocking due to the use of the Shared lock itself, but merely a technical aspect, not semantically visible to the program. It is equivalent to the operating system pausing the thread, a techical

[issue8800] add threading.RWLock

Sebastian Noack added the comment: I've added a new patch, that implements a shared/exclusive lock as described in my comments above, for the threading and multiprocessing module. -- Added file: http://bugs.python.org/file27350/Added-ShrdExclLock-to-threading-and-multiprocessing.patch

[issue8800] add threading.RWLock

Sebastian Noack added the comment: Using a lock as context manager is the same as calling lock.acquire(blocking=True) and it will in fact block while waiting for an other thread to release the lock. In your code, the internal lock is indeed just hold for a very short period of time while acqui

[issue8800] add threading.RWLock

Changes by Christian Heimes : -- versions: +Python 3.4 -Python 3.2 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: Excellent points. For 3. however, I don't consider access to synchronized state to be "blocking". Blocking means to block while waiting for the lock. The internal lock is never held for any amount of time. Perhaps I'll cook up a new patch with these t

[issue8800] add threading.RWLock

Sebastian Noack added the comment: I would love to see a reader/writer lock implementation shipped with Python's threading (and multiprocessing) module. But I have some issues with the patch: 1. I would avoid the terms 'read' and 'write' as those terms are referring just to one of many use cas

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: > Perhaps this proposed patch is overly complex. I don't know. Do you think relaxing the semantics would make the implementation significantly faster? (interesting link, btw) -- ___ Python tracker

[issue8800] add threading.RWLock

Kristján Valur Jónsson added the comment: I should add that on Windows, the new SRW that is part of Vista and Windows 7, uses locking, that is it favors neither readers or writers. It appears that nowadays the complex semantics of RWLocks have not really proven worthwile. See http://msdn.mic

[issue8800] add threading.RWLock

Changes by Andrew Svetlov : -- nosy: +asvetlov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue8800] add threading.RWLock

Changes by Matthew Lauber : -- nosy: +mklauber ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue8800] add threading.RWLock

Jeffrey Yasskin added the comment: You're definitely right that posix and java don't make these usable from the normal lock API. And I don't think it's essential that they be usable as RLocks, although it's nice for Conditions. I think what I'm actually saying is that, if you have an acquire(

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: > In this case, "acquire" isn't ambiguous. All the other lock types > actually acquire a write lock, so it makes sense to have the operation > with the same name they use also acquire a write lock on this object. Well, it looks like a strange kind of "consisten

[issue8800] add threading.RWLock

Jeffrey Yasskin added the comment: In this case, "acquire" isn't ambiguous. All the other lock types actually acquire a write lock, so it makes sense to have the operation with the same name they use also acquire a write lock on this object. I realized that read/write locks are actually share

[issue8800] add threading.RWLock

Antoine Pitrou added the comment: I'm not sure it's a good idea to have a default context manager, and default acquire/release methods. "In the face of ambiguity, refuse the temptation to guess". Is there any use in being able to use a RWLock inside a condition variable? Docstrings should be

[issue8800] add threading.RWLock

Jeffrey Yasskin added the comment: This patch doesn't apply cleanly against the py3k tree. Since Python 2.7 is in beta, and there's no 2.8, this can only go into python 3, so you should work against that tree. It's a bit annoying that the R in RWLock stands for a different word from in RLock

[issue8800] add threading.RWLock

Changes by Antoine Pitrou : -- nosy: +jyasskin stage: -> patch review versions: +Python 3.2 -Python 2.7 ___ Python tracker ___ ___ Pyt

[issue8800] add threading.RWLock

New submission from Kristján Valur Jónsson : Provided is a patch, implementing a RWLock using a ConditionVariable and Lock. It has rdlock() and wrlock() methods, and an unlock() method, similar to the rwlock API in pthreads. It has context managers rdlocked() and wrlocked(). In addition, it conf