On 20 Apr, 18:32, Brian <knair...@yahoo.com> wrote: > On Apr 20, 9:18 am, alessiogiovanni.bar...@gmail.com wrote: > > > > > On 20 Apr, 17:03, Brian <knair...@yahoo.com> wrote: > > > > I'm using the third-party "processing" module in Python 2.5, which may > > > have become the "multiprocessing" module in Python 2.6, to speed up > > > the execution of a computation that takes over a week to run. The > > > relevant code may not be relevant, but it is: > > > > q1, q2 = processing.Queue(), processing.Queue() > > > p1 = processing.Process(target=_findMaxMatch, args= > > > (reciprocal, user, clusters[1:(numClusters - 1)/2], questions, > > > copy.copy(maxMatch), q1)) > > > p2 = processing.Process(target=_findMaxMatch, args= > > > (reciprocal, user, clusters[(numClusters - 1)/2:], questions, copy.copy > > > (maxMatch), q2)) > > > p1.start() > > > p2.start() > > > maxMatch1 = q1.get()[0] > > > maxMatch2 = q2.get()[0] > > > p1.join() > > > p2.join() > > > if maxMatch1[1] > maxMatch2[1]: > > > maxMatch = maxMatch1 > > > else: > > > maxMatch = maxMatch2 > > > > This code just splits up the calculation of the cluster that best > > > matches 'user' into two for loops, each in its own process, rather > > > than one. (It's not important what the cluster is.) > > > > The error I get is: > > > > [21661.903889] Out of memory: kill process 14888 (python) score 610654 > > > or a child > > > [21661.903930] Killed process 14888 (python) > > > Traceback (most recent call last): > > > ...etc. etc. ... > > > > Running this process from tty1, rather than GNOME, on my Ubuntu Hardy > > > system allowed the execution to get a little further than under GNOME. > > > > The error was surprising because with just 1 GB of memory and a single > > > for loop I didn't run into the error, but with 5 GB and two processes, > > > I do. I believe that in the 1 GB case there was just a lot of > > > painfully slow swapping going on that allowed it to continue. > > > 'processing' appears to throw its hands up immediately, instead. > > > > Why does the program fail with 'processing' but not without it? Do you > > > have any ideas for resolving the problem? Thanks for your help. > > > If your program crashes with more of one process, maybe you handle the > > Queue objects > > not properly? If you can, post the code of _findMaxMatch. > > Thanks for your interest. Here's _findMaxMatch: > > def _findMaxMatch(reciprocal, user, clusters, sources, maxMatch, > queue): > for clusternumminusone, cluster in enumerate(clusters): > clusterFirstData, clusterSecondData = cluster.getData(sources) > aMatch = gum.calculateMatchGivenData(user.data, None, None, > None, user2data=clusterSecondData)[2] > if reciprocal: > maxMatchB = gum.calculateMatchGivenData(clusterFirstData, > None, None, None, user2data=user.secondUserData)[2] > aMatch = float(aMatch + maxMatchB) / 2 > if aMatch > maxMatch[1]: > maxMatch = [clusternumminusone + 1, aMatch] > queue.put([maxMatch])
You can post the entire error message + full traceback? -- http://mail.python.org/mailman/listinfo/python-list
