On May 21, 11:13 am, "A.T.Hofkamp" <[EMAIL PROTECTED]> wrote: > On 2008-05-21, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > > I'd appreciate any help. I've got a list of files in a directory, and > > I'd like to iterate through that list and process each one. Rather > > than do that serially, I was thinking I should start five threads and > > process five files at a time. > > > Is this a good idea? I picked the number five at random... I was > > Depends what you are doing. > If you are mainly reading/writing files, there is not much to gain, since 1 > process will already push the disk IO system to its limit. If you do a lot of > processing, then more threads than the number of processors is not much use. > If > you have more 'burtsy' behavior (first do lot of reading, then lot of > processing, then again reading, etc), then the system may be able to do some > scheduling and keep both the processors and the file system busy. > > I cannot really give you advice on threading, I have never done that. You may > want to consider an alternative, namely multi-tasking at OS level. If you can > easily split the files over a number of OS processes (written in Python), you > can make the Python program really simple, and let the OS handle the > task-switching between the programs. > > Sincerely, > Albert
Albert, Thanks for your response - I appreciate your time! I am mainly reading and writing files, so it seems like it might not be a good idea. What if I read the whole file into memory first, and operate on it there? They are not large files... Either way, I'd hope that someone might respond with an example, as then I could test and see which is faster! Thanks again. -- http://mail.python.org/mailman/listinfo/python-list
