Python GIL problem is solved you can look at here multiprocessing module was developed, which enables true parallel processing with Python on a multiprocore machine, with an interface very close to that of the threading module.Moreover, one can run a program across machines! In other words, the multiprocessing module allowsto run several threads not only on the different cores of one machine, but on many machines at once, in cooperation in the same manner that threads cooperate on one machine. By the way, this idea is similarto something I did for Perl at IIT BOMBAY 15 dayz ago (PerlDSM: A Distributed Shared Memory System for Perl.Proceedings of PDPTA 2002, 63-68). We will not cover the cross-machine case here
1#dice probability finder 2# based on Python multiprocessing class 3 # usage: python DiceProb.py n k nreps nthreads 4 # where we wish to find the probability of getting a total of k dots 5 # when we roll n dice; we’ll use nreps total repetitions of the 6 # simulation, dividing those repetitions among nthreads threads 7 8 import sys 9 import random 10 from multiprocessing import Process, Lock, Value 11 12 class glbls: # globals, other than shared 13 n = int(sys.argv[1]) 14 k = int(sys.argv[2]) 15 nreps = int(sys.argv[3]) 16 nthreads = int(sys.argv[4]) 17 thrdlist = [] # list of all instances of this class 18 19 def worker(id,tot,totlock): 20 mynreps = glbls.nreps/glbls.nthreads 21 r = random.Random() # set up random number generator 22 count = 0 # number of times get total of k 23 for i in range(mynreps): 24 if rolldice(r) == glbls.k: 25 count += 1 26 totlock.acquire() 27 tot.value += count 28 totlock.release() 29 # check for load balance 30 print ’thread’, id, ’exiting; total was’, count 31 32 def rolldice(r): 33 ndots = 0 34 for roll in range(glbls.n): 35 dots = r.randint(1,6) 36 ndots += dots 37 return ndots 38 39 def main(): 40 tot = Value(’i’,0) 70 CHAPTER 3. THE PYTHON THREADS AND MULTIPROCESSING MODULES 41 totlock = Lock() 42 for i in range(glbls.nthreads): 43 pr = Process(target=worker, args=(i,tot,totlock)) 44 glbls.thrdlist.append(pr) 45 pr.start() 46 for thrd in glbls.thrdlist: thrd.join() 47 # adjust for truncation, in case nthreads doesn’t divide nreps evenly 48 actualnreps = glbls.nreps/glbls.nthreads * glbls.nthreads 49 print ’the probability is’,float(tot.value)/actualnreps 50 51 if __name__ == ’__main__’: 52 main() how does it work? The general structure looks similar to that of the Python threading module, using Process() to create a create a thread, start() to get it running, Lock() to create a lock, acquire() and release() to lock and unlock a lock, and so on now look for multiprocessing 1 #!/usr/bin/env python 2 3 # prime number counter, based on Python multiprocessing class 4 5 # usage: python PrimeThreading.py n nthreads 6 # where we wish the count of the number of primes from 2 to n, and to 7 # use nthreads to do the work 8 9 # uses Sieve of Erathosthenes: write out all numbers from 2 to n, then 10 # cross out all the multiples of 2, then of 3, then of 5, etc., up to 11 # sqrt(n); what’s left at the end are the primes 12 13 import sys 14 import math 15 from multiprocessing import Process, Lock, Array, Value 16 17 class glbls: # globals, other than shared 18 n = int(sys.argv[1]) 19 nthreads = int(sys.argv[2]) 20 thrdlist = [] # list of all instances of this class 21 22 def prmfinder(id,prm,nxtk,nxtklock): 23 lim = math.sqrt(glbls.n) 24 nk = 0 # count of k’s done by this thread, to assess load balance 25 while 1: 26 # find next value to cross out with 27 nxtklock.acquire() 28 k = nxtk.value 29 nxtk.value = nxtk.value + 1 30 nxtklock.release() 31 if k > lim: break 32 nk += 1 # increment workload data 33 if prm[k]: # now cross out 34 r = glbls.n / k 35 for i in range(2,r+1): 36 prm[i*k] = 0 37 print ’thread’, id, ’exiting; processed’, nk, ’values of k’ 38 39 def main(): 40 prime = Array(’i’,(glbls.n+1) * [1]) # 1 means prime, until find otherwise 41 nextk = Value(’i’,2) # next value to try crossing out with 42 nextklock = Lock() 43 for i in range(glbls.nthreads): 44 pf = Process(target=prmfinder, args=(i,prime,nextk,nextklock)) 45 glbls.thrdlist.append(pf) 46 pf.start() 47 for thrd in glbls.thrdlist: thrd.join() 48 print ’there are’, reduce(lambda x,y: x+y, prime) - 2, ’primes’ 49 50 if __name__ == ’__main__’: 51 main() look here i use array () now may i know you got it or you can use the Pool class to create a set of threads, rather than doing so “by hand” in a loop as above.You can start them with various initial values for the threads using Pool.map(), which works similarly toPython’s ordinary map() function On Tue, Feb 8, 2011 at 2:33 PM, Noufal Ibrahim <nou...@gmail.com> wrote: > On Tue, Feb 08 2011, Baishampayan Ghose wrote: > > >> Mutliprocessing means, data copying, talking to each other through > PIPES, > >> also it has its issues with running on Windows (all function calls > should be > >> pickelable) > >> > >> Threads seems pretty stable on most platforms where Python runs. > > > > Threads won't help you much because of the Python GIL. > > [...] > > He's suggesting an alternate implementation which might work around the > serialisation enforced by the GIL for CPU bound threads. > > -- > _______________________________________________ > BangPypers mailing list > BangPypers@python.org > http://mail.python.org/mailman/listinfo/bangpypers > _______________________________________________ BangPypers mailing list BangPypers@python.org http://mail.python.org/mailman/listinfo/bangpypers
