On Feb 21, 6:25 pm, Tino Wildenhain <t...@wildenhain.de> wrote: > intellimi...@gmail.com wrote: > > I wrote a script to process textual data and extract phrases from > > them, storing these phrases in a dictionary. It encounters a > > MemoryError when there are about 11.18M keys in the dictionary, and > > the size is about 1.5GB. I tried multiple times, and the error occurs > > everytime at exactly the same place (with the same number of keys in > > the dict). I then split the dictionary into two using a simple > > algorithm: > > > if str[0]<='m': > > dict=dict1 > > else: > > dict=dict2 > > > #use dict... > > > And it worked fine. The total size of the two dictionaries well > > exceeded 2GB yet no MemoryError occured. > > > I have 1GB of pysical memory and 3GB in pagefile. Is there a limit to > > the size or number of entries that a single dictionary can possess? By > > searching on the web I can't find a clue why this problem occurs. > > From what can be deducted from the headers of your message: > X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1;... > you are using windows? > It seems either python or windows memory management somehow prevent > the use of continuous memory areas that large. > We've got such an example somewhere down the list which was similar > (iirc it was a large string in memory) which runned perfectly > with linux. You can try yourself maybe by installing ubuntu > on the same host. (If you feel fit you can even skip the install > and run it off life CD but then you need to fiddle a little to > get swap space on disk) > > Regards > Tino > > > -- > >http://mail.python.org/mailman/listinfo/python-list > > > > smime.p7s > 4KViewDownload
Yes, it's winxp, I forgot to mention it. Thanks for the reply. -- http://mail.python.org/mailman/listinfo/python-list
