Re: Real-world use of concurrent.futures

2014-05-13 Thread Andrew McLean
On 08/05/2014 21:44, Ian Kelly wrote: > On May 8, 2014 12:57 PM, "Andrew McLean" > wrote: > > So far so good. However, I thought this would be an opportunity to > > explore concurrent.futures and to see whether it offered any benefits > > over the more explicit approach

Re: Real-world use of concurrent.futures

2014-05-08 Thread Chris Angelico
On Fri, May 9, 2014 at 5:45 AM, Marko Rauhamaa wrote: > Chris Angelico : > >> Before you go too far down roads that are starting to look >> problematic: A DNS lookup is a UDP packet out and a UDP packet in >> (ignoring the possibility of TCP queries, which you probably won't be >> doing here). May

Re: Real-world use of concurrent.futures

2014-05-08 Thread Andrew McLean
On 08/05/2014 21:44, Ian Kelly wrote: > I don't think it needs to be "messy". Something like this should do > the trick, I think: > > from concurrent.futures import * > from itertools import islice > > def batched_pool_runner(f, iterable, pool, batch_size): > it = iter(iterable) > # Submit the

Re: Real-world use of concurrent.futures

2014-05-08 Thread Ian Kelly
On May 8, 2014 12:57 PM, "Andrew McLean" wrote: > So far so good. However, I thought this would be an opportunity to > explore concurrent.futures and to see whether it offered any benefits > over the more explicit approach discussed above. The problem I am having > is that all the discussions I ca

Re: Real-world use of concurrent.futures

2014-05-08 Thread Terry Reedy
On 5/8/2014 2:55 PM, Andrew McLean wrote: I have a problem that would benefit from a multithreaded implementation and having trouble understanding how to approach it using concurrent.futures. The details don't really matter, but it will probably help to be explicit. I have a large CSV file that

Re: Real-world use of concurrent.futures

2014-05-08 Thread Marko Rauhamaa
Andrew McLean : > That's fine, but I suspect that isn't a helpful pattern if I have a > very large number of tasks. In my case I could run out of memory if I > tried submitting all of the tasks to the executor before processing > any of the results. This is related to flow control. You'll need an

Re: Real-world use of concurrent.futures

2014-05-08 Thread Marko Rauhamaa
Chris Angelico : > Before you go too far down roads that are starting to look > problematic: A DNS lookup is a UDP packet out and a UDP packet in > (ignoring the possibility of TCP queries, which you probably won't be > doing here). Maybe it would be easier to implement it as asynchronous > networ

Re: Real-world use of concurrent.futures

2014-05-08 Thread Andrew McLean
On 08/05/2014 20:06, Chris Angelico wrote: > On Fri, May 9, 2014 at 4:55 AM, Andrew McLean wrote: >> Because of latency in the DNS lookup this could >> benefit from multithreading. > Before you go too far down roads that are starting to look > problematic: A DNS lookup is a UDP packet out and a UD

Re: Real-world use of concurrent.futures

2014-05-08 Thread Chris Angelico
On Fri, May 9, 2014 at 4:55 AM, Andrew McLean wrote: > Because of latency in the DNS lookup this could > benefit from multithreading. Before you go too far down roads that are starting to look problematic: A DNS lookup is a UDP packet out and a UDP packet in (ignoring the possibility of TCP queri