On Wed, Jul 1, 2009 at 7:48 AM, Barak, Ron<ron.ba...@lsi.com> wrote: > Hi, > > I think I'm up against a limitation in cStringIO.StringIO(), but could not > find any clues on the web, especially not in > http://docs.python.org/library/stringio.html. > > What I have is the following function: > > def concatenate_files(self,filename_array): > import types > > f = cStringIO.StringIO() > for filename in filename_array: > log_stream = LogStream(filename) > if not isinstance(log_stream, types.NoneType): > try: > string_ = log_stream.input_file.read() > except AttributeError, e: > sys.stderr.write("AttributeError: "+str(e)+"\n") > sys.stderr.write("log_stream: > "+str(log_stream)+"\n") > else: > return(None) > > f.write(string_) > return (f) > > And, when the list of files - in filename_array - is pointing to long/large > enough files, I get the exception: > > Traceback (most recent call last): > File "svm_ts_tool_in_progress.py", line 244, in OnSelectCell > self.InspectorViewController(row,col) > File "svm_ts_tool_in_progress.py", line 1216, in InspectorViewController > self.InspectorePaneGetRecords(self.req_table, rec) > File "svm_ts_tool_in_progress.py", line 1315, in InspectorePaneGetRecords > log_stream = > ConcatenatedLogStream(self.events_dict[req_table]["db_dict"].keys()) > File "c:\Documents and > Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 31, in > __init__ > self.input_file = self.concatenate_files(filename_array) > File "c:\Documents and > Settings\rbarak\rbarak_devel\dpm16\ConcatenatedLogStream.py", line 47, in > concatenate_files > string_ = log_stream.input_file.read() > MemoryError > > Anyone knows whet's the limitation on cStringIO.StringIO() objects ? > Could you suggest a better way to create a string that contains the > concatenation of several big files ? >
MemoryErrors are caused because, for whatever reason, the OS won't allocate any more memory for the process. For a 32-bit Windows process, that maximum is 2 GB. No matter what programming language or library you use, you can't bypass that limit. There is a way to up the limit to 3GB but I don't know how to do it. The link below has all the information about Windows memory limits. If you need lots of really big strings in memory simultaneously, you'll have to get a 64-bit system with a ton of RAM. http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx > Thanks, > Ron. > > -- > http://mail.python.org/mailman/listinfo/python-list > > -- http://mail.python.org/mailman/listinfo/python-list
