> > > 1. Anyone knows whet's the limitation on cStringIO.StringIO() objects ? > 2. Could you suggest a better way to create a string that contains the > concatenation of several big files ? > > This isn't a limit in cStringIO.StringIO, but a limit on the amount of memory a process is able to allocate from the OS. There's no real way around it-- but you can take a different approach.
You have to get the memory somewhere, but if what you're combining can't fit into memory (generally, 2 GB, but it depends on OS and platform), you can use memory from a different source-- the hard-drive for instance. I would suggest you create a temporary file somewhere, and instead of reading the contents into memory, read it in chunks and write it to this temporary file, then go on and do the rest. That way you'll never have so much in memory that you hit the limits. When this is done, you can move this temporary file to where you want it, or read from it and pass its output somewhere. --S
-- http://mail.python.org/mailman/listinfo/python-list
