[issue9260] A finer grained import lock

Changes by Antoine Pitrou : -- status: pending -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://

[issue9260] A finer grained import lock

Changes by Antoine Pitrou : -- status: open -> pending ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue9260] A finer grained import lock

Changes by Antoine Pitrou : -- stage: patch review -> committed/rejected status: pending -> open ___ Python tracker ___ ___ Python-bugs

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: I have now pushed the patch. -- resolution: -> fixed status: open -> pending ___ Python tracker ___ ___

[issue9260] A finer grained import lock

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue9260] A finer grained import lock

Roundup Robot added the comment: New changeset edb9ce3a6c2e by Antoine Pitrou in branch 'default': Issue #9260: A finer-grained import lock. http://hg.python.org/cpython/rev/edb9ce3a6c2e -- nosy: +python-dev ___ Python tracker

[issue9260] A finer grained import lock

Brett Cannon added the comment: I don't feel the need to, but I can in a few days if you want me to (just let me know if you do). -- ___ Python tracker ___ _

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: Does anyone else want to review this patch? -- ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: I had forgotten to tackle threadless builds, this patch fixes it. -- Added file: http://bugs.python.org/file25521/module_locks9.patch ___ Python tracker ___

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: Updated patch against tip. I also changed the internal API of module locks a bit (acquire() raises _DeadlockError instead of returning False, and deadlock detection is not optional anymore). -- Added file: http://bugs.python.org/file25496/module_locks

[issue9260] A finer grained import lock

Changes by Andrew Svetlov : -- nosy: +asvetlov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue9260] A finer grained import lock

Martin v. Löwis added the comment: The patch parser of Rietveld actually choked on the git binary diff. It now skips over these chunks. -- ___ Python tracker ___ ___

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: Updated patch against tip, and with a comment of what deadlock avoidance does (in _ModuleLock.acquire's docstring). -- Added file: http://bugs.python.org/file25475/module_locks7.patch ___ Python tracker

[issue9260] A finer grained import lock

Martin v. Löwis added the comment: Can you please elaborate in the patch what the deadlock avoidance does? AFAICT, the comment explains that it is able to detect deadlocks, but nowhere says what it does when it has detected a deadlock. Also, please submit patches against default's head, or st

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: > Now, if x and y are executed in separate threads - won't it deadlock? Well, the patch has a deadlock avoidance mechanism, and it includes unit tests for precisely this situation. I cannot promise the algorithm is perfect (although there *are* a bunch of tests

[issue9260] A finer grained import lock

Martin v. Löwis added the comment: I still wonder whether Graham Dumpleton's observation has merits. Suppose we have these modules # a.py time.sleep(10) import b # b.py time.sleep(10) import a # main.py def x(): import a def y(): import b Now, if x and y are executed in separate threads

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: Updated patch with a couple new tests. -- stage: -> patch review Added file: http://bugs.python.org/file25471/module_locks6.patch ___ Python tracker __

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: Updated patch also adds unit tests for the module locks and the deadlock avoidance algorithm. -- Added file: http://bugs.python.org/file25470/module_locks5.patch ___ Python tracker

[issue9260] A finer grained import lock

Changes by Eric Snow : -- nosy: +eric.snow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue9260] A finer grained import lock

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file25469/module_locks4.patch ___ Python tracker ___ ___ Python-bugs-list mailin

[issue9260] A finer grained import lock

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file25466/module_locks4.patch ___ Python tracker ___ ___ Python-bugs-list mail

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: Updated patch also makes PyImport_ImportModuleNoBlock a simple alias of PyImport_ImportModule. -- Added file: http://bugs.python.org/file25466/module_locks4.patch ___ Python tracker

[issue9260] A finer grained import lock

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file25399/module_locks3.patch ___ Python tracker ___ ___ Python-bugs-list mailin

[issue9260] A finer grained import lock

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file25398/module_locks3.patch ___ Python tracker ___ ___ Python-bugs-list mail

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: Updated patch fixes the performance issue and disposes of module locks when they aren't used anymore. Only the afterfork question remains. Should I hook in threading's own facility? Should we wait for an atfork module? Something else. -- Added file: h

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: New patch gets rid of the reliance on _thread.RLock (uses non-recursive locks instead), and should solve the synchronization issue. Other issues remain. -- Added file: http://bugs.python.org/file25394/module_locks2.patch __

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: Ok, here is a draft patch for the new importlib. Several issues with this patch: - introduces a pure Python function (_lock_unlock_module) on the fast import path - synchronization issues due to interruptibility of pure Python code (see _ModuleLock.acquire) -

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: I believe this new patch should be much more robust than the previous one. It also adds a test for the improvement (it freezes an unpatched interpreter). -- Added file: http://bugs.python.org/file24114/implock5.patch ___

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: > If, on the other hand, we really want to reduce the number of cases > where a deadlock would occur by increasing the locking granularity, > then it's the way to go. Yes, that's the point. Today you have to be careful when mixing imports and threads. The probl

[issue9260] A finer grained import lock

Charles-François Natali added the comment: > That's true. Do you think temptatively acquiring the lock (without > blocking) would solve the issue? I think it should work. Something along those lines: while True: if lock.acquire(0): lock.tstate = tstate return True else:

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: > It owns the lock, but hasn't yet updated the lock's owner > (lock->tstate), so another thread calling detect_circularity() will > think that this lock is available, and will proceed, which can > eventually lead to a deadlock. That's true. Do you think temptat

[issue9260] A finer grained import lock

Charles-François Natali added the comment: IIUC, the deadlock avoidance code just checks that acquiring a per-module lock won't create a cycle. However, I think there's a race, because the cycle detection and the lock acquisition is not atomic. For example, let's say we have a thread exactly he

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: New prototype with per-module import locks and deadlock avoidance. When a deadlock due to threaded circular imports is detected, the offending import returns the partially constructed module object (as would happen in single-threaded mode). Probably lacks a t

[issue9260] A finer grained import lock

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue9260] A finer grained import lock

Brett Cannon added the comment: I'll have a look when I can (hopefully EuroPython). -- ___ Python tracker ___ ___ Python-bugs-list mai

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: > That's my point; loaders are using the lock implicitly so that's why > we don't need to worry about the global import lock just for path > hooks. It seems like you are suggesting using the global import lock > purely for compatibility, and what I am saying is

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: > Graham Dumpleton added the comment: > > How is this going to deal with cyclical imports where different > threads could import at the same time different modules within that > cycle? I need to look through the proposed patch and work out exactly > what it do

[issue9260] A finer grained import lock

Brett Cannon added the comment: That's my point; loaders are using the lock implicitly so that's why we don't need to worry about the global import lock just for path hooks. It seems like you are suggesting using the global import lock purely for compatibility, and what I am saying is that lo

[issue9260] A finer grained import lock

Graham Dumpleton added the comment: How is this going to deal with cyclical imports where different threads could import at the same time different modules within that cycle? I need to look through the proposed patch and work out exactly what it does, but am concerned about whether this appro

[issue9260] A finer grained import lock

Changes by Alexander Belopolsky : Removed file: http://bugs.python.org/file18005/unnamed ___ Python tracker ___ ___ Python-bugs-list mailing li

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: > So changing the locking mechanism most likely won't break loaders > because they are not using the current import lock anyway and so > already have their own issues. Are you sure they aren't using it implicitly? In vanilla py3k, PyImport_ImportModuleLevel()

[issue9260] A finer grained import lock

Brett Cannon added the comment: What I'm saying is that loaders are quite possibly not thread-safe already, so we don't need to do any special for them. If you look at PEP 302 you will notice not a single mention of loaders needing to care about the import lock because there is no mention of

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: > That's why I said we should supply a context decorator (or function) which > will handle the lock appropriately, taking the name of the module to import > as an argument so the locking is fine-grained. Ok, so what are you saying is that we can break compatibi

[issue9260] A finer grained import lock

Brett Cannon added the comment: On Wed, Jul 14, 2010 at 12:34, Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > > > So I say we don't worry about loaders being thread-safe. If __import__ > > handles the locking for a specific module then it will hold the lock > > on behalf of the

[issue9260] A finer grained import lock

Antoine Pitrou added the comment: > So I say we don't worry about loaders being thread-safe. If __import__ > handles the locking for a specific module then it will hold the lock > on behalf of the loader. Yes but what happens if two different modules are imported from two different threads, and

[issue9260] A finer grained import lock

Brett Cannon added the comment: So I say we don't worry about loaders being thread-safe. If __import__ handles the locking for a specific module then it will hold the lock on behalf of the loader. Now if someone decides to call load_module on their own, that's there business, but they should

[issue9260] A finer grained import lock

Changes by Alexander Belopolsky : -- nosy: +belopolsky ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue9260] A finer grained import lock

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file17999/implock.patch ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue9260] A finer grained import lock

Changes by Antoine Pitrou : Removed file: http://bugs.python.org/file17998/implock.patch ___ Python tracker ___ ___ Python-bugs-list mailing li

[issue9260] A finer grained import lock

New submission from Antoine Pitrou : This is an implementation of the idea suggested in: http://mail.python.org/pipermail/python-dev/2003-February/033445.html The patch creates a dictionary of reentrant locks keyed by module full name. Trying to import a module or package will first get the loc