Re: large dictionary creation takes a LOT of time.

2005-04-30 Thread Kent Johnson
Maksim Kasimov wrote: sorry for my question, but i've read the documentation, and can't find where is the explanation of how it is exactly works (but of course i do believe you). If it is buit in function, can i see the source code of the method to find it out? Kent Johnson wrote: http://docs.p

Re: large dictionary creation takes a LOT of time.

2005-04-30 Thread Maksim Kasimov
sorry for my question, but i've read the documentation, and can't find where is the explanation of how it is exactly works (but of course i do believe you). If it is buit in function, can i see the source code of the method to find it out? Kent Johnson wrote: Maksim Kasimov wrote: Kent Johnson w

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread possibilitybox
oh, right, i did only one eighth to check and see if it was scaling near linearly, as i couldn't even run profiling without python dying. i have 400mb ram and 2ghz processor, on freebsd, so it shouldn't be performance. i'll try your suggestions and see how it works. -- http://mail.python.org/ma

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread R. C. James Harlow
On Friday 29 April 2005 11:53, Ville Vainio wrote: > > "Kent" == Kent Johnson <[EMAIL PROTECTED]> writes: > > Kent> if frequency.has_key(word): > Kent> frequency[word] += 1 > Kent> else: > Kent> frequency[word] = 1 > > This is a good place to use 'get' method of dict: > > freque

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread Caleb Hattingh
In fact, as one of the Peter's (either Otten or Hansen) explained to me, for line in open(file): is actually both faster (being buffered) and generally better for very large files because it doesn't read the whole file into memory, like readlines does (if you have a memory limitation). On Fri,

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread Kent Johnson
Maksim Kasimov wrote: Kent Johnson wrote: > for line in open(path): the line of your example raise another question: opened file will be read at once time, as method readlines() do, or it will be read line by line as method readline() do. It will be read line by line as readline() does. as far i

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread Maksim Kasimov
Kent Johnson wrote: Here is a little cleaner version. It takes about a second to run on my PC. What hardware are you running on? path = 'DonQuixote.txt' frequency = {} for line in open(path): for word in line.split(): if frequency.has_key(word): frequency[word] += 1

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread Roy Smith
In article <[EMAIL PROTECTED]>, "possibilitybox" <[EMAIL PROTECTED]> wrote: > this code here: > > > def wordcount(lines): > for i in range(len(lines)/8): > words = lines[i].split(" ") > if not locals().has_key("frequency"): > frequency = {} > for word in

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread bearophileHUGS
Ville Vainio: > This is a good place to use 'get' method of dict: > frequency[word] = frequency.get(word,0) + 1 I think Kent Johnson is longer, but a bit faster... Bye, Bearophile -- http://mail.python.org/mailman/listinfo/python-list

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread Ville Vainio
> "Kent" == Kent Johnson <[EMAIL PROTECTED]> writes: Kent> if frequency.has_key(word): Kent> frequency[word] += 1 Kent> else: Kent> frequency[word] = 1 This is a good place to use 'get' method of dict: frequency[word] = frequency.get(word,0) + 1 -- Ville Vainio http://tin

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread Kent Johnson
possibilitybox wrote: this code here: def wordcount(lines): for i in range(len(lines)/8): words = lines[i].split(" ") if not locals().has_key("frequency"): frequency = {} for word in words: if frequency.has_key(word): frequency[wor

Re: large dictionary creation takes a LOT of time.

2005-04-29 Thread Paul Rubin
"possibilitybox" <[EMAIL PROTECTED]> writes: > this code here: > > > def wordcount(lines): > for i in range(len(lines)/8): > words = lines[i].split(" ") > if not locals().has_key("frequency"): > frequency = {} > for word in words: > if frequenc

large dictionary creation takes a LOT of time.

2005-04-28 Thread possibilitybox
this code here: def wordcount(lines): for i in range(len(lines)/8): words = lines[i].split(" ") if not locals().has_key("frequency"): frequency = {} for word in words: if frequency.has_key(word): frequency[word] += 1