On Wed, Jan 7, 2009 at 3:22 PM, Amit k. Saha <[email protected]> wrote: > On Wed, Jan 7, 2009 at 3:10 PM, Sibtey Mehdi <[email protected]> wrote: >> Hi, >> >> >> >> I use multiprocessing to compare more then one set of files. >> >> For comparison each set of files (i.e. Old file1 Vs New file1) I create a >> process, >> >> Process(target=compare, args=(oldFile, newFile)).start() >> >> It takes 61 seconds execution time. >> >> >> >> When I do the same comparison without implementing multiprocessing, it takes >> 52 seconds execution time. >> >> >> >> The parallel processing time should be lesser. >> >> >> >> I am not able to get advantage of multiprocessing here.
No surprise here. A process is a heavyweight entity. The design of spawning a process per file comparison won't scale and as the number of comparisons increase, will asymptotically perform worse than doing all in a single process/thread. The flaw is in the design. A parallel programming task as above is about distributing m jobs onto n resources. There are two trivial cases for it -> where n is 1 (m jobs in 1 resource, i.e a single process/thread) and where m is 1 (i.e 1 job per resource - spawing a resource for every job as you are doing). These two trivial cases are two extremes and will always yield poorer results when compared to choosing the right value of m and n. In this case you should group your comparison tasks and execute say for example 10 comparisons per thread/process. Design a thread/process pool to scale your solution. This is simply Concurrent/Parallel Programming 101... > > Depends on the overheads of spawning multiple threads/processes v$ > your actual processing. The same way, things are compared in old > school when they used to compare macros v$ functions in C/C++. > > -Amit > > >> >> >> >> Any suggestions can be very helpful. >> >> >> >> Thanks, >> >> Gopal >> >> >> >> >> >> >> >> _______________________________________________ >> BangPypers mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/bangpypers >> >> > > > > -- > Amit Kumar Saha > http://amitksaha.blogspot.com > http://amitsaha.in.googlepages.com/ > Skype: amitkumarsaha > _______________________________________________ > BangPypers mailing list > [email protected] > http://mail.python.org/mailman/listinfo/bangpypers > -- -Anand _______________________________________________ BangPypers mailing list [email protected] http://mail.python.org/mailman/listinfo/bangpypers
