Re: [go-nuts] Idiomatic distribute-process-collect pattern

2023-05-22 Thread Brian Candler
Isn't Ian's solution basically the same, but using a Waitgroup instead of an Errgroup? I think the Fetch() is just a placeholder for "this is where the implementation detail of the task would go" On Sunday, 21 May 2023 at 21:54:39 UTC+1 Andrew Harris wrote: > The recipe on 10:26 should work bu

Re: [go-nuts] Idiomatic distribute-process-collect pattern

2023-05-21 Thread Andrew Harris
The recipe on 10:26 should work but it's definitely contrived - using Fetch looks strange to me, and the cancellation behavior of errgroup is another moving part that adds some overhead. If it's sufficient, Ian's solution groks better :) On Sunday, May 21, 2023 at 7:38:31 AM UTC-7 Tobias Klaus

Re: [go-nuts] Idiomatic distribute-process-collect pattern

2023-05-21 Thread Tobias Klausmann
Hi! First off, thanks both Ian and Andrew! On Sat, 20 May 2023, Ian Lance Taylor wrote: > On Sat, May 20, 2023 at 1:23 AM Tobias Klausmann > wrote: > > What I wonder about is the collection stage: the assumption that there > > will be exactly as many results as inputs seems brittle, and the

Re: [go-nuts] Idiomatic distribute-process-collect pattern

2023-05-20 Thread Ian Lance Taylor
On Sat, May 20, 2023 at 1:23 AM Tobias Klausmann wrote: > > What I wonder about is the collection stage: the assumption that there > will be exactly as many results as inputs seems brittle, and the only > real way to handle errors is to put them into the result struct and > examining them

Re: [go-nuts] Idiomatic distribute-process-collect pattern

2023-05-20 Thread Andrew Harris
For aggregating results, an alternative to blocking on outstanding payloads is blocking on outstanding workers. You might find this talk interesting, and it has a pretty clever solution for "outstanding workers": https://about.sourcegraph.com/blog/go/gophercon-2018-rethinking-classical-concurren

[go-nuts] Idiomatic distribute-process-collect pattern

2023-05-20 Thread Tobias Klausmann
Hi! I find myself often writing tools that follow the pattern in the subject, that is: - Gather some input to process, e.g. a list of filenames from the command line. If this can be streamed (processing starting before knowing all inputs) even better, but not strictly necessary. - Start N wo