Re: gather information from various files efficiently

2004-12-15 Thread Paul McGuire
"Klaus Neuner" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Hello, > > I need to gather information that is contained in various files. > > Like so: > > file1: > = > foo : 1 2 > bar : 2 4 > baz : 3 > = > > file2: > = > foo

Thanks (Re: gather information from various files efficiently)

2004-12-15 Thread Klaus Neuner
Hello, (Sorry for beginning a new thread. Google does not allow me to reply to my own posting. And I cannot use a newsreader at the moment.) Thanks to all who participated in the thread. I tried the try- and the if-solution. The setdefault-solution didn't work in my program. With the try-soluti

Re: gather information from various files efficiently

2004-12-14 Thread Mike Meyer
Simon Brunning <[EMAIL PROTECTED]> writes: > On Tue, 14 Dec 2004 10:40:56 -0500, Peter Hansen <[EMAIL PROTECTED]> wrote: >> Keith Dart wrote: >> > Sigh, this reminds me of a discussion I had at my work once... It seems >> > to write optimal Python code one must understand various probabilites of >

Re: gather information from various files efficiently

2004-12-14 Thread Jeff Shannon
Klaus Neuner wrote: Yet, I have got 43 such files. Together they are 4,1M large. In the future, they will probably become much larger. At the moment, the process takes several hours. As it is a process that I have to run very often, I would like it to be faster. Others have shown how you can m

Re: gather information from various files efficiently

2004-12-14 Thread Steven Bethard
Klaus Neuner wrote: The straightforward way to solve this problem is to create a dictionary. Like so: [...] a, b = get_information(line) if a in dict.keys(): dict[a].append(b) else: dict[a] = [b] So I timed the three suggestions with a few different datasets: > cat builddict.py def askpermi

Re: gather information from various files efficiently

2004-12-14 Thread Simon Brunning
On Tue, 14 Dec 2004 10:40:56 -0500, Peter Hansen <[EMAIL PROTECTED]> wrote: > Keith Dart wrote: > > Sigh, this reminds me of a discussion I had at my work once... It seems > > to write optimal Python code one must understand various probabilites of > > your data, and code according to the likely sc

Re: gather information from various files efficiently

2004-12-14 Thread Fernando Perez
Keith Dart wrote: > Aye... > > the dict.keys() line creates a temporary list, and then the 'in' does a > linear search of the list. Better would be: > > try: > dict[a].append(b) > except KeyError: > dict[a] = [b] > > since you expect the key to be there most of the time, this method i

Re: gather information from various files efficiently

2004-12-13 Thread Peter Hansen
Keith Dart wrote: Sigh, this reminds me of a discussion I had at my work once... It seems to write optimal Python code one must understand various probabilites of your data, and code according to the likely scenario. And this is different from optimizing in *any* other language in what way? -Pet

Re: gather information from various files efficiently

2004-12-13 Thread Keith Dart
Fredrik Lundh wrote: ... if dct.has_key(a): dct[a].append(b) else: dct[a] = [b] the drawback here is that if the number of collisions are high, you end up doing lots of extra dictionary lookups. in that case, there are better ways to do this. Sigh, this reminds me of a discussion I had at m

Re: gather information from various files efficiently

2004-12-13 Thread Richie Hindle
[Keith] > Sigh, this reminds me of a discussion I had at my work once... It seems > to write optimal Python code one must understand various probabilites of > your data, and code according to the likely scenario. 8-) s/Python //g -- Richie Hindle [EMAIL PROTECTED] -- http://mail.python.org/

Re: gather information from various files efficiently

2004-12-13 Thread Fredrik Lundh
Keith Dart wrote: >>> try: >>> dict[a].append(b) >>> except KeyError: >>> dict[a] = [b] the drawback here is that exceptions are relatively expensive; if the number of collisions are small, you end up throwing and catching lots of exceptions. in that case, there are better ways to do thi

Re: gather information from various files efficiently

2004-12-13 Thread Keith Dart
Kent Johnson wrote: Keith Dart wrote: try: dict[a].append(b) except KeyError: dict[a] = [b] or my favorite Python shortcut: dict.setdefault(a, []).append(b) Kent Hey, when did THAT get in there? ;-) That's nice. However, the try..except block is a useful pattern for many similiar situ

Re: gather information from various files efficiently

2004-12-13 Thread Kent Johnson
Keith Dart wrote: try: dict[a].append(b) except KeyError: dict[a] = [b] or my favorite Python shortcut: dict.setdefault(a, []).append(b) Kent -- http://mail.python.org/mailman/listinfo/python-list

gather information from various files efficiently

2004-12-13 Thread Klaus Neuner
Hello, I need to gather information that is contained in various files. Like so: file1: = foo : 1 2 bar : 2 4 baz : 3 = file2: = foo : 5 bar : 6 baz : 7 = file3: = foo : 4 18 bar : 8 ===

Re: gather information from various files efficiently

2004-12-13 Thread Keith Dart
Klaus Neuner wrote: Hello, I need to gather information that is contained in various files. Like so: file1: = foo : 1 2 bar : 2 4 baz : 3 = file2: = foo : 5 bar : 6 baz : 7 = file3: = foo : 4 18 bar