On Mar 7, 10:13 am, calcp...@aol.com wrote: > from multiprocessing import Pool > ... > pool = Pool(processes=2) # start 2 worker processes > << > > Wow, cool, is this part of parallel python? Does this only work on a > multi-core PC or can this be made to work over a cluster as well?
This is standard library's multiprocessing library: http://docs.python.org/library/multiprocessing.html It comes with Python >= 2.6 default. There are backports for 2.4 and 2.5 at http://pypi.python.org/pypi/multiprocessing/2.6.2.1 This module is for harnessing multiple cores. For cluster parallelism I would suggest you to give a try to IPython (http://ipython.scipy.org/ doc/stable/html/overview.html#interactive-parallel-computing) To me the multiprocessing module is the simplest approach to utilize the second lazy core in my laptop. (Intel Core 2 Duo 2.5G, 4GiB memory). I will use this example to demonstrate how quickly one can write code distributed on multiple cores in Python. In my local Sage v4.3.3 installation, in notebook using 1 million element list from timeit import default_timer as clock def f(x): return x**3 + x**2 + x t1 = clock() map(f, range(1000000)) t2 = clock() print "Elapsed time using single process:", t2-t1 Elapsed time using single process: 4.46871089935 from timeit import default_timer as clock from multiprocessing import Pool def f(x): return x**3 + x**2 + x if __name__ == '__main__': t1 = clock() pool = Pool(processes=2) # start 2 worker processes dummy = pool.map(f, range(1000000)) t2 = clock() print "Elapsed time using two processes:", t2-t1 Elapsed time using two processes: 3.15281605721 The gain is ~1.4x. In this example increasing the list length the ratio can go upto 1.7x. when I run the same test on sagenb.org Elapsed time using single process: 0.937326192856 Elapsed time using two processes: 0.970607995987 Heh single process execution takes less time :) If I try with 10 million element then I get this error. Traceback (click to the left of this block for traceback) ... MemoryError Maybe someone can give me a smarter example to demonstrate multiprocessing in Sage-notebook without failing on the main sagenb.org server. -- To post to this group, send email to sage-support@googlegroups.com To unsubscribe from this group, send email to sage-support+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-support URL: http://www.sagemath.org
