I don’t have an example but a concurrent solution seems straightforward to
reason about. Here’s a description of where I’d start:
Perhaps have a type to send on the buffered chan from the one producer
goroutine?
type Chunk struct {
Index int
Data []byte
}
Then make any number of worker goroutines that read from the buffered chan
Chunk (maybe a count equal to https://golang.org/pkg/runtime/#NumCPU) and
do the processing. Then have a single consumer goroutine that reads
processed Chunks from these workers via an unbuffered channel. This
consumer orders the work output and returns the data once complete.
Alternatively the workers could write their output into a slice or array
where each index can only be written by one goroutine.
The best solution depends on your application. For a network server with
multiple clients a serial approach may be simpler than a concurrent one and
have similar performance. How will your batch processing be used?
Matt
On Tuesday, January 23, 2018 at 8:03:25 AM UTC-6, Peng Yu wrote:
>
> Hi,
>
> I'd like to implement batch processing in go. There should be a goroutine
> for reading input and several worker goroutines for processing the input in
> chunks. The workers may not finish processing the chunks in the same order
> as they are in the input. But the processing results should be output in
> the same order as the chunks in the input.
>
> I see a number of resources that are relevant to the above goal. But I am
> not sure what is the best solution in terms of how easy the code is and how
> much less performance overhead there is. Does anyone have any advice on the
> best solution to this problem? Thanks.
>
> P.S., here is one webpage, but it does not allow multiple workers.
>
>
> https://blog.drkaka.com/batch-get-from-golangs-buffered-channel-9638573f0c6e
> --
> Regards,
> Peng
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
