On 2019-05-23, Chris Angelico <ros...@gmail.com> wrote: > On Fri, May 24, 2019 at 5:37 AM Bob van der Poel <b...@mellowood.ca> wrote: >> >> I've got a short script that loops though a number of files and >> processes them one at a time. I had a bit of time today and figured >> I'd rewrite the script to process the files 4 at a time by using 4 >> different instances of python. My basic loop is: >> >> for i in range(0, len(filelist), CPU_COUNT): >> for z in range(i, i+CPU_COUNT): >> doit( filelist[z]) >> >> With the function doit() calling up the program to do the >> lifting. Setting CPU_COUNT to 1 or 5 (I have 6 cores) makes no >> difference in total speed. I'm processing about 1200 files and my >> total duration is around 2 minutes. No matter how many cores I use >> the total is within a 5 second range. > > Where's the part of the code that actually runs them across multiple > CPUs? Also, are you spending your time waiting on the disk, the CPU, > IPC, or something else?
He said he's using N differenct Python instances, and he even provided the code that runs in each instance which is obviously processesing 1/Nth of the files. It's a pretty good bet that I/O is the limiting factor. -- Grant -- https://mail.python.org/mailman/listinfo/python-list
