[issue7060] test_multiprocessing dictionary changed size errors and hang

STINNER Victor added the comment: > RuntimeError: dictionary changed size during iteration I didn't see this bug the last 5 years, so I consider that the issue has been fixed in the meanwhile :-) -- resolution: out of date -> fixed status: open -> closed

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: Okay, let's say it is fixed. Adding Richard to nosy so that he can review the issue if he's interested. -- resolution: -> out of date stage: needs patch -> committed/rejected ___ Python tracker

[issue7060] test_multiprocessing dictionary changed size errors and hang

STINNER Victor added the comment: I never seen this issue, can we close it? #7105 has been fixed in Python 3, not in Python 2.7. -- nosy: +haypo ___ Python tracker ___ __

[issue7060] test_multiprocessing dictionary changed size errors and hang

Changes by Ask Solem : -- nosy: +asksol ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/m

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: I don't think this is a blocker anymore. Kristján's patch has been ported and the test_multiprocessing freezes have been "fixed" by adding a timeout. As for the original purpose of this ticket, issue7105 is still (part of?) the answer. -- priority: rele

[issue7060] test_multiprocessing dictionary changed size errors and hang

Kristján Valur Jónsson added the comment: Ah, okay then. Let's focus on solving 7414 then. -- ___ Python tracker ___ ___ Python-bugs-

[issue7060] test_multiprocessing dictionary changed size errors and hang

Kristján Valur Jónsson added the comment: See my analysis from issue 7314: The other processes are all waiting to ReadFile(). Maybe they have yet to run to the point where they open the socket to which the master process is trying to connect to. The mystery is also: Why does the harmless act

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: > Yes, this is quite different. > After seeing this, I created issue 7314. > I didn't realize that it had anything to do with my patch but rather > blamed it on my brand new windows 7. I don't think it has anything to do with your patch. While diagnosing the p

[issue7060] test_multiprocessing dictionary changed size errors and hang

Kristján Valur Jónsson added the comment: Yes, this is quite different. After seeing this, I created issue 7314. I didn't realize that it had anything to do with my patch but rather blamed it on my brand new windows 7. -- ___ Python tracker

[issue7060] test_multiprocessing dictionary changed size errors and hang

Changes by Antoine Pitrou : -- dependencies: +multiprocessing should not wait endlessly ___ Python tracker ___ ___ Python-bugs-list mai

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: Ok, the problem is that a finalizer calls _decref which tries to open an XMLRPC connection (!) to an address on which nobody listens anymore. Then stupid SocketClient (in multiprocessing.connection) loops endlessly because it is designed to do so when it gets EC

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: It seems the error is different, though. This is what I get after Ctrl-C (I omit the tracebacks from the other processes): Traceback (most recent call last): File "/home/antoine/py3k/__svn__/Lib/test/test_multiprocessing.py", line 1071, in test_number_of_obje

[issue7060] test_multiprocessing dictionary changed size errors and hang

R. David Murray added the comment: It is. TestZZZNumberOfObjects hangs at r76238, and does not hang at r76237 on my Gentoo Linux box. -- ___ Python tracker ___ _

[issue7060] test_multiprocessing dictionary changed size errors and hang

Changes by R. David Murray : -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/o

[issue7060] test_multiprocessing dictionary changed size errors and hang

R. David Murray added the comment: It is. TestZZZNumberOfObjects hangs at r76237, and does not hang at r76238 on my Gentoo Linux box. -- ___ Python tracker ___ _

[issue7060] test_multiprocessing dictionary changed size errors and hang

Changes by R. David Murray : -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/o

[issue7060] test_multiprocessing dictionary changed size errors and hang

R. David Murray added the comment: It is. TestZZZNumberOfObjects hangs at r76238, and does not hang at r76239 on my Gentoo Linux box. -- ___ Python tracker ___ _

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: I don't know if it's your last change, but test_multiprocessing is now hanging on all buildbots. -- ___ Python tracker ___ _

[issue7060] test_multiprocessing dictionary changed size errors and hang

Kristján Valur Jónsson added the comment: Thanks. Fixed in revision 76239. -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: > Kristján Valur Jónsson added the comment: > > Ok, I ported the change again in revision 76238, removing the traceback > from the stored exception object. "versionchanged" should be 3.2, not 3.1. Thanks! -- ___

[issue7060] test_multiprocessing dictionary changed size errors and hang

Kristján Valur Jónsson added the comment: Ok, I ported the change again in revision 76238, removing the traceback from the stored exception object. -- ___ Python tracker ___ ___

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: Please note there's a patch in issue7105 to allow safer iteration of weak dicts. -- ___ Python tracker ___ _

[issue7060] test_multiprocessing dictionary changed size errors and hang

Kristján Valur Jónsson added the comment: I can easily port it again, and remove the traceback from the exception. But won't we be sweeping that GC problem under the carpet then? -- ___ Python tracker _

[issue7060] test_multiprocessing dictionary changed size errors and hang

Benjamin Peterson added the comment: If this is not ported by the time we get to a 2.7 or 3.2 release, I'd like to back it out. -- priority: normal -> release blocker ___ Python tracker

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: Kristján: > So, can one just clear the __traceback__ member? Yes, or use `exc_value.with_traceback(None)`. -- ___ Python tracker ___ __

[issue7060] test_multiprocessing dictionary changed size errors and hang

Changes by Jon Parise : -- nosy: +jon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mai

[issue7060] test_multiprocessing dictionary changed size errors and hang

Changes by Antoine Pitrou : -- dependencies: +weak dict iterators are fragile because of unpredictable GC runs ___ Python tracker ___ _

[issue7060] test_multiprocessing dictionary changed size errors and hang

Kristján Valur Jónsson added the comment: I didn't realize that the traceback was attached to the exception object in py3k. This makes the use of such objects more dangerous because of the circular references. The recommended practice of exc_type, exc_obj = sys.exc_info()[:2] is precisely t

[issue7060] test_multiprocessing dictionary changed size errors and hang

Benjamin Peterson added the comment: I disagree about reverting it. We have this bug report, and that change is sitting on the merge queue again... -- ___ Python tracker ___

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: > The unittest only keeps an exception _object_ around, not the associated > traceback. In py3k, the traceback is attached to the exception (on the __traceback__ attribute) ;) -- ___ Python tracker

[issue7060] test_multiprocessing dictionary changed size errors and hang

Kristján Valur Jónsson added the comment: The unittest only keeps an exception _object_ around, not the associated traceback. There should be no frames and thus reference cycles associated with this. I agree that the problem should be tackled, rather than swept under the carpet like a revert

[issue7060] test_multiprocessing dictionary changed size errors and hang

Antoine Pitrou added the comment: It was a bad idea to revert the change, because we may now forget about the problem rather than fix it. It is likely that the "dictionary changed size during iteration" occurs because a GC collection gets triggered during the iteration on the weakdict. A GC col

[issue7060] test_multiprocessing dictionary changed size errors and hang

Kristján Valur Jónsson added the comment: Interesting. I don't have a linux machine to debug this. My patch appears harmless enough. We are only storing an exception _object_, not any tracebacks or such. If this were happening on my windows machine I would put in breakpoints where the igno

[issue7060] test_multiprocessing dictionary changed size errors and hang

R. David Murray added the comment: Benjamin tried reverting the weakref patch, but that didn't fix it. Turns out the problem is the other patch in that merge, that adds saving of the exception to AssertRaises. Adding Kristjan as that was his patch. (The tests pass with the weakref patch applie

[issue7060] test_multiprocessing dictionary changed size errors and hang

Jesse Noller added the comment: It's passing for me on trunk/py3k at least on OS/X - I'm going to need to dredge up a linuxvm -- ___ Python tracker ___ _

[issue7060] test_multiprocessing dictionary changed size errors and hang

Jesse Noller added the comment: Adding gps - I haven't had a chance to grok the changes, and me getting to a fix is going to take a few. -- nosy: +gregory.p.smith ___ Python tracker ___

[issue7060] test_multiprocessing dictionary changed size errors and hang

R. David Murray added the comment: r75241 seems to be the cause. Tests passes on r75240, but gives the dict error and hang on r75241. -- ___ Python tracker ___ _

[issue7060] test_multiprocessing dictionary changed size errors and hang

Changes by R. David Murray : -- components: +Library (Lib) ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:

[issue7060] test_multiprocessing dictionary changed size errors and hang

New submission from R. David Murray : test_multiprocessing is producing tracebacks and hanging on py3k trunk. I think this started happening fairly recently, as it seemed to work in one not-too-old checkout until I did an svn up and then it started failing. 3.1 seems fine. I've attached the tr