Re: [go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Louki Sumirniy
ah yes, no, if you see the code in the play link below, it only has three channels, ops, done and ready. I just figured out that I replaced that ready by putting the close in the clause that processes incoming ops, and it's unused as well. I managed to trim it down to just one channel, the ops

Re: [go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Louki Sumirniy
oh, I did forget one thing. The race detector does not flag a race in this code: https://play.golang.org/p/M1uGq1g4vjo (play refuses to run it though) As I understand it, that's because the add/subtract operations are happening serially within the main handler goroutine. I suppose if I were to

Re: [go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Louki Sumirniy
I more or less eventually figured that out since it is impossible to query the number of workers without a race anyway, and then I started toying with atomic.Value and made that one race as well (obviously the value was copied by fmt.Println). I guess keeping track of the number of workers is on

Re: [go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Robert Engels
Channels use sync primitives under the hood so you are not saving anything by using multiple channels instead of a single wait group. > On May 2, 2019, at 5:57 PM, Louki Sumirniy > wrote: > > As I mentioned earlier, I wanted to see if I could implement a waitgroup with > channels instead of

[go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Louki Sumirniy
As I mentioned earlier, I wanted to see if I could implement a waitgroup with channels instead of the stdlib's sync.Atomic counters, and using a special type of concurrent datatype called a PN Converged Replicated Datatype. Well, I'm not sure if this implementation precisely implements this typ

[go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Øyvind Teig
Thanks for the reference to Dave Cheney's blog note! And for this thread, quite interesting to read. I am not used to explicitly closing channels at all (occam (in the ninetees) and XC (now)), but I have sat through several presentations on conferences seen the theme being discussed, like with t

Re: [go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Louki Sumirniy
Ah, so this is what they are for - the same thing implemented with channels would be a nasty big slice with empty struct quit channels to first tell the main they are done. wg.Done() and wg.Wait() eliminate the complexity that a pure channel implementation would require. With that code I also

Re: [go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Steven Hartland
You can see it doesn't wait by adding a counter as seen here: https://play.golang.org/p/-eqKggUEjhQ On 02/05/2019 21:09, Louki Sumirniy wrote: I have been spending my time today getting my knowledge of this subject adequate enough to use channels for a UDP transport with FEC creating sharded pi

Re: [go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Steven Hartland
Without the wait group it doesn't wait, so you're not guaranteed for all / any of the goroutines to complete. On 02/05/2019 21:09, Louki Sumirniy wrote: I have been spending my time today getting my knowledge of this subject adequate enough to use channels for a UDP transport with FEC creating

[go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Louki Sumirniy
I have been spending my time today getting my knowledge of this subject adequate enough to use channels for a UDP transport with FEC creating sharded pieces of the packets, and I just found this and played with some of the code on it and I just wanted to mention these things: https://dave.chene

[go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Louki Sumirniy
It's not precisely the general functionality that I will implement for my transport, but here is a simple example of a classifier type processing queue: https://play.golang.org/p/ytdrXgCdbQH This processes a series of sequential integers and pops them into an array to find the highest factor o

[go-nuts] Re: using channels to gather bundles from inputs for batch processing

2019-05-02 Thread Louki Sumirniy
Yeah, I was able to think a bit more about it as I was falling asleep later and I realised how I meant it to run. I had to verify that indeed channels are FIFO queues, as that was the basis of this way of using them. The receiver channel is unbuffered, and lives in one goroutine. When it receiv