Thank you every one, I ended up using a solution similar to what Gary Herron suggested : Caching the output to a list of lists, one per file, and only doing the IO when the list reaches a certain treshold. After playing around with the list threshold I ended up with faster execution times than originally and while having a maximum of two files open at a time! Its only a matter of trading memory for open files. It could be that using this strategy with asynchronous IO or threads could yield even faster times, but I haven't tested it. Again, much appreciated thanks for all your suggestions.
Andre M. Descombes > Hello, > > I need to split a very big file (10 gigabytes) into several thousand > smaller files according to a hash algorithm, I do this one line at a > time. The problem I have is that opening a file using append, writing > the line and closing the file is very time consuming. I'd rather have > the files all open for the duration, do all writes and then close them > all at the end. > The problem I have under windows is that as soon as I get to 500 files I > get the Too many open files message. I tried the same thing in Delphi > and I can get to 3000 files. How can I increase the number of open files > in Python? > > Thanks in advance for any answers! > > Andre M. Descombes -- http://mail.python.org/mailman/listinfo/python-list
