intellimi...@gmail.com wrote:
I wrote a script to process textual data and extract phrases from them, storing these phrases in a dictionary. It encounters a MemoryError when there are about 11.18M keys in the dictionary, and the size is about 1.5GB. I tried multiple times, and the error occurs everytime at exactly the same place (with the same number of keys in the dict). I then split the dictionary into two using a simple algorithm:if str[0]<='m': dict=dict1 else: dict=dict2 #use dict... And it worked fine. The total size of the two dictionaries well exceeded 2GB yet no MemoryError occured. I have 1GB of pysical memory and 3GB in pagefile. Is there a limit to the size or number of entries that a single dictionary can possess? By searching on the web I can't find a clue why this problem occurs.
From what can be deducted from the headers of your message: X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1;... you are using windows? It seems either python or windows memory management somehow prevent the use of continuous memory areas that large. We've got such an example somewhere down the list which was similar (iirc it was a large string in memory) which runned perfectly with linux. You can try yourself maybe by installing ubuntu on the same host. (If you feel fit you can even skip the install and run it off life CD but then you need to fiddle a little to get swap space on disk) Regards Tino
-- http://mail.python.org/mailman/listinfo/python-list
smime.p7s
Description: S/MIME Cryptographic Signature
-- http://mail.python.org/mailman/listinfo/python-list
