Re: Multi thread reading a file

In message <025ff4f1$0$20657$c3e8...@news.astraweb.com>, Steven D'Aprano wrote: > On Sun, 05 Jul 2009 12:12:22 +1200, Lawrence D'Oliveiro wrote: > >> In message >> <1beffd94-cfe6-4cf6-bd48-2ccac8637...@j32g2000yqh.googlegroups.com>, ryles >> wrote: >> >> # Oh... yeah. I really *did* want '

Re: Multi thread reading a file

On Sun, 05 Jul 2009 12:12:22 +1200, Lawrence D'Oliveiro wrote: > In message <1beffd94-cfe6-4cf6- > bd48-2ccac8637...@j32g2000yqh.googlegroups.com>, ryles wrote: > > # Oh... yeah. I really *did* want 'is None' and not '== None' which > # iter() will do. Sorry guys! >> >> Please don't let

Re: Multi thread reading a file

In message <1beffd94-cfe6-4cf6- bd48-2ccac8637...@j32g2000yqh.googlegroups.com>, ryles wrote: # Oh... yeah. I really *did* want 'is None' and not '== None' which # iter() will do. Sorry guys! > > Please don't let this happen to you too ;) Strange. others have got told off for using "==

Re: Multi thread reading a file

On Jul 2, 11:55 pm, Paul Rubin wrote: > Yeah, it should allow supplying a predicate instead of using == on > a value.  How about (untested): > >    from itertools import * >    ... >    for row in takewhile(lambda x: x is sentinel, >                          starmap(i

Re: Multi thread reading a file

"Gabriel Genellina" writes: > We're talking about the iter() builtin behavior, and that uses == > internally. Oh, I see. Drat. > It could have used an identity test, and that would be better for this > specific case. But then iter(somefile.read, '') wouldn't work. Yeah, it should allow supplyi

Re: Multi thread reading a file

En Fri, 03 Jul 2009 00:15:40 -0300, > escribió: ryles writes: >    sentinel = object() I agree, this is cleaner than None. We're still in the same boat, though, regarding iter(). Either it's 'item == None' or 'item == object ()' Use "item is sentinel". We're talking about the iter() bui

Re: Multi thread reading a file

ryles writes: > >    sentinel = object() > > I agree, this is cleaner than None. We're still in the same boat, > though, regarding iter(). Either it's 'item == None' or 'item == object ()' Use "item is sentinel". -- http://mail.python.org/mailman/listinfo/python-list

Re: Multi thread reading a file

On Jul 2, 10:20 pm, Paul Rubin wrote: > ryles writes: > > >>> # Oh... yeah. I really *did* want 'is None' and not '== None' > > >>> which iter() will do. Sorry guys! > > > Please don't let this happen to you too ;) > > None is a perfectly good value to put onto a que

Re: Multi thread reading a file

ryles writes: > >>> # Oh... yeah. I really *did* want 'is None' and not '== None' > >>> which iter() will do. Sorry guys! > > Please don't let this happen to you too ;) None is a perfectly good value to put onto a queue. I prefer using a unique sentinel to mark the end of the stream: sentin

Re: Multi thread reading a file

On Jul 2, 6:10 am, "Gabriel Genellina" wrote: > En Wed, 01 Jul 2009 12:49:31 -0300, Scott David Daniels   > escribió: > > These loops work well with the two-argument version of iter, > > which is easy to forget, but quite useful to have in your bag > > of tricks: > > >      def convert(in_queue,

Re: Multi thread reading a file

En Wed, 01 Jul 2009 12:49:31 -0300, Scott David Daniels escribió: Gabriel Genellina wrote: ... def convert(in_queue, out_queue): while True: row = in_queue.get() if row is None: break # ... convert row out_queue.put(converted_line) These loops work well with the two-argum

Re: Multi thread reading a file

Mag Gam wrote: > On Wed, Jul 1, 2009 at 12:54 AM, Gabriel Genellina wrote: >> Does the format conversion involve a significant processing time? If not, >> the total time is dominated by the I/O time (reading and writing the file) >> so it's doubtful you gain anything from multiple threads. > > The

Re: Multi thread reading a file

LOL! :-) Why not. I think I will take just do it single thread for now and if performance is really that bad I can re investigate. Either way, thanks everyone for your feedback! I think I like python a lot because of the great user community and wiliness to help! On Wed, Jul 1, 2009 at 1:07 AM

Re: Multi thread reading a file

Thanks for the response Gabriel. On Wed, Jul 1, 2009 at 12:54 AM, Gabriel Genellina wrote: > En Tue, 30 Jun 2009 22:52:18 -0300, Mag Gam escribió: > >> I am very new to python and I am in the process of loading a very >> large compressed csv file into another format.  I was wondering if I >> ca

Re: Multi thread reading a file

In message , Mag Gam wrote: > I am very new to python and I am in the process of loading a very > large compressed csv file into another format. I was wondering if I > can do this in a multi thread approach. Why bother? -- http://mail.python.org/mailman/listinfo/python-list

Re: Multi thread reading a file

Gabriel Genellina wrote: ... def convert(in_queue, out_queue): while True: row = in_queue.get() if row is None: break # ... convert row out_queue.put(converted_line) These loops work well with the two-argument version of iter, which is easy to forget, but quite useful to have

Re: Multi thread reading a file

Gabriel Genellina wrote: > En Tue, 30 Jun 2009 22:52:18 -0300, Mag Gam escribió: > >> I am very new to python and I am in the process of loading a very >> large compressed csv file into another format. I was wondering if I >> can do this in a multi thread approach. > > Does the format conversio

Re: Multi thread reading a file

En Tue, 30 Jun 2009 22:52:18 -0300, Mag Gam escribió: I am very new to python and I am in the process of loading a very large compressed csv file into another format. I was wondering if I can do this in a multi thread approach. Does the format conversion involve a significant processing time

Multi thread reading a file

Hello All, I am very new to python and I am in the process of loading a very large compressed csv file into another format. I was wondering if I can do this in a multi thread approach. Here is the pseudo code I was thinking about: Let T = Total number of lines in a file, Example 100 (1 mil