On Sun, May 26, 2019 at 11:05 AM Grant Edwards <grant.b.edwa...@gmail.com> wrote:
> On 2019-05-23, Chris Angelico <ros...@gmail.com> wrote: > > On Fri, May 24, 2019 at 5:37 AM Bob van der Poel <b...@mellowood.ca> > wrote: > >> > >> I've got a short script that loops though a number of files and > >> processes them one at a time. I had a bit of time today and figured > >> I'd rewrite the script to process the files 4 at a time by using 4 > >> different instances of python. My basic loop is: > >> > >> for i in range(0, len(filelist), CPU_COUNT): > >> for z in range(i, i+CPU_COUNT): > >> doit( filelist[z]) > >> > >> With the function doit() calling up the program to do the > >> lifting. Setting CPU_COUNT to 1 or 5 (I have 6 cores) makes no > >> difference in total speed. I'm processing about 1200 files and my > >> total duration is around 2 minutes. No matter how many cores I use > >> the total is within a 5 second range. > > > > Where's the part of the code that actually runs them across multiple > > CPUs? Also, are you spending your time waiting on the disk, the CPU, > > IPC, or something else? > > He said he's using N differenct Python instances, and he even provided > the code that runs in each instance which is obviously processesing > 1/Nth of the files. > > It's a pretty good bet that I/O is the limiting factor. > > I did say that ... I also retracted it. In my first eg. I was still running one process at a time ... no multi-task at all. Parallel is up and running and is very, very fast ... loads my CPUs to 100% > > > > -- > https://mail.python.org/mailman/listinfo/python-list > -- **** Listen to my FREE CD at http://www.mellowood.ca/music/cedars **** Bob van der Poel ** Wynndel, British Columbia, CANADA ** EMAIL: b...@mellowood.ca WWW: http://www.mellowood.ca -- https://mail.python.org/mailman/listinfo/python-list
